# The Program Testing Ability of Large Language Models for Code

Weimin Xiong1,2, Yiwen Guo3 , Hao Chen4
1Tencent Security Big Data Lab, 2Peking University, 3Independent Researcher,
4UC Davis
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding
author

###### Abstract

Recent development of large language models (LLMs) for code like CodeX and
CodeT5+ demonstrates tremendous promise in achieving code intelligence. Their
ability of synthesizing code that completes a program for performing a pre-
defined task has been intensively tested and verified on benchmark datasets
including HumanEval and MBPP. Yet, evaluation of these LLMs from more
perspectives (than just program synthesis) is also anticipated, considering
their broad scope of applications in software engineering. In this paper, we
explore the ability of LLMs for testing programs/code. By performing thorough
analyses of recent LLMs for code in program testing, we show a series of
intriguing properties of these models and demonstrate how program testing
ability of LLMs can be improved. Following recent work which utilizes
generated test cases to enhance program synthesis, we further leverage our
findings in improving the quality of the synthesized programs and show +11.77%
and +4.22% higher code pass rates on HumanEval+ comparing with the
GPT-3.5-turbo baseline and the recent state-of-the-art, respectively. Our code
is available at https://github.com/WeiminXiong/TestingLLM.

##  1 Introduction

The community has witnessed a surge in the development of large language
models (LLMs), which have achieved incredible ability in understanding and
generating not only texts but also code. LLMs for code (CodeX (Chen et al.,
2021), StarCoder (Li et al., 2023b), CodeT5+ (Wang et al., 2023b), etc) have
been widely adopted to a variety of applications to achieve code intelligence.
However, current evaluation of these LLMs mostly focuses on program
completion/synthesis, despite the models can also be utilized in other
applications. As the field continues to advance, evaluation of these models
from more perspectives is anticipated, which could facilitate deeper
understanding of the LLMs.

As the core of analyzing the behavior of code, the ability of generating
proper test cases is of great desire to software engineering. Although
embryonic development of using deep models in testing has been shown (Tufano
et al., 2020; 2022), with the remarkable progress in LLMs, it is unclear how
far have such abilities of AI been advanced when these powerful models are
equipped. In this paper, we, for the first time, analyze the ability of recent
LLMs in testing programs/code. Our analyses are performed based on 164
problems from HumanEval+ (Chen et al., 2021) and 427 sanitized problems from
MBPP (Austin et al., 2021). We consider 4 test-case generation settings (i.e.,
self-generated, all-generated, oracle, and placeholder in Figure 1) and test a
collection of 11 competitive LLMs for code (including 4 LLMs that have around
1 billion parameters and 7 substantially larger LLMs). We conducted a variety
of experiments, from which many intriguing takeaway messages are delivered.

Several very recent papers (Shi et al., 2022; Li et al., 2023a; Chen et al.,
2023) have shown that appropriate usage of even generated test cases can
improve the quality of program synthesise, in a spirit that the synthesized
programs that could pass a large number of test cases are more likely to be
correct. Nevertheless, the quality of the generated test cases largely impacts
the performance of such methods. Due to the lack of systematic evaluation of
the testing ability of LLMs for code, it is unclear how to craft test cases
that could be potentially more helpful to program synthesis and, more broadly,
code intelligence. The studies in this paper aim to shed light on this. We
will demonstrate that, substantially improved program synthesis performance
can be obtained by utilizing takeaway messages in our studies. Specifically,
on GPT-3.5-turbo, we can achieve +11.77% higher code pass rate on HumanEval+,
in comparison with the GPT-3.5-turbo baseline. When compared with a very
recent state-of-the-art called CodeT, our solution achieves +4.22% higher code
pass rate.

##  2 Evaluation Metrics

To make the evaluation more reliable and comprehensive, it is crucial to first
design some suitable metrics, like BLEU (Papineni et al., 2002), ROUGE (Lin,
2004), and the pass rate (Chen et al., 2021) for evaluating machine
translation, text summarization, and program synthesis, respectively. In this
section, we specify two main evaluation metrics to evaluate the program
testing ability of LLMs, from the perspective of correctness and diversity.

Pass rate  In software engineering, we expect test cases to represent some
desired “ground-truth” functionality of the tested program/code. In practice,
such “ground-truth” functionality can be described in the header comments of a
function (i.e., docstrings of the function) and tested using the oracle
implementation, as in HumanEval (Chen et al., 2021) and MBPP Austin et al.
(2021). The oracle program/code should be able to pass the test, if a
generated test case is correct. Therefore, we leverage the pass rate as a
measure to evaluate the correctness of the generated test cases. For a fair
comparison, we instruct each model to generate three test cases in the prompt,
and, when a model generates more than three test cases, we select the first
three for evaluation. Assuming that there are in total $M$ programming
problems in an experimental dataset and, for each problem, we have $N$
program/code implementations to be generated test cases for. Each model has
only one chance to generate these test cases for each program/code. Then, we
calculate the pass rate as:

$P=\frac{1}{MN}\sum_{i=1}^{M}\sum_{j=1}^{N}\frac{p_{ij}}{n_{ij}},$ (1)

where $n_{ij}$ is the number of test cases in $\mathcal{Q}_{ij}$ which
includes no more than three test cases generated for the $j$-th program/code
implementation of the $i$-th problem by the evaluated LLM at once, i.e.,
$\mathcal{Q}_{ij}=\\{(x_{ijk},y_{ijk})\\}_{k}$, and $p_{ij}$ is the number of
test cases (in $\mathcal{Q}_{ij}$) that do not fail the oracle.

The pass rate defined in Eq. (1) measures correctness of the generated test
cases. However, as can be seen in Figure 1, the model can generate duplicate
test cases that are less helpful, even though they are correct. To avoid such
an evaluation bias, we further advocate deduplication in the set of test cases
that are considered as correct, which leads to computation of a deduplicated
pass rate defined as $P^{\prime}=\frac{1}{MN}\sum\sum
p^{\prime}_{ij}/n^{\prime}_{ij}$, where we use ′ to denote the numbers of
unique test cases.

Coverage rate  In addition to the above pass rates, we further consider
coverage rate as a more fine-grained metric for evaluating the diversity of
the generated test cases. According to its definition, converge rate computes
the degree to which the code is executed, given a test case. Since, for each
program/code, we keep no more than three test cases at once, we calculate how
much percentage of the control structure is covered given these test cases.
Similar to Eq. (1), we evaluate the performance of testing all programs/code
over all $M\times N$ times of generation, i.e., we calculate

$C=\frac{1}{MN}\sum_{i=1}^{M}\sum_{j=1}^{N}c_{ij}.$ (2)

We utilize the pytest 111https://pytest.org library to evaluate branch
coverage for all the three test cases for each code and aggregate the results
for all programs/code and all problems. Apparently, a higher $C$ indicates
better testing ability of an LLM, since we expect all parts of the
programs/code to be executed to find our all potential bugs, with the set of
test cases generated by this LLM.

##  3 Large Language Models for Code

In this section, we outline the evaluated models. We adopt some “small” models
whose numbers of parameters are around 1B (to be more specific, from 770M to
1.3B in our choices) and some larger models that achieve state-of-the-art
performance in the task of program synthesis.

For the small models, we use InCoder (1.3B) (Fried et al., 2023), CodeGen2
(1B) (Nijkamp et al., 2023a), CodeT5+ (770M) (Wang et al., 2023b), and
SantaCoder (1.1B) (Allal et al., 2023). InCoder is a unified generative model
that can perform program/code synthesis as well as code editing, and it
combines the strengths of causal language modeling and masked language
modeling. The CodeGen2 model was trained on a deduplicated subset of the Stack
v1.1 dataset (Kocetkov et al., 2023), and its training is formatted with a
mixture of objectives for causal language modeling and span corruption.
CodeT5+ is an encoder-decoder model trained on several pre-training tasks
including span denoising and two variants of causal language modeling.
SantaCoder was trained on the Python, Java, and JavaScript code in the Stack
dataset. The pass rate (Chen et al., 2021) of programs generated by these
models is compared in Table 1. When evaluating the (program) pass rate, we let
the model generate 200 code implementations for each problem, and we set the
temperature to 0.2, 0.6, and 0.8 for calculating pass@1, pass@10, and
pass@100, respectively.

As for larger models that achieve state-of-the-art program synthesis
performance, we use CodeGen2 (16B) (Nijkamp et al., 2023a), CodeGen-Multi
(16B) Nijkamp et al. (2023b), CodeGen-Mono (16B) Nijkamp et al. (2023b),
StarCoder (15B) (Li et al., 2023b), WizardCoder (15B) (Luo et al., 2023),
CodeGeeX2 (6B) (Zheng et al., 2023), and GPT-3.5-turbo. CodeGen-Multi and
CodeGen-Mono are two large models from the first version of CodeGen. CodeGen-
Multi was first trained on the pile dataset (Gao et al., 2020) and then
trained on a subset of the publicly available BigQuery dataset which contains
code written in C, C++, Go, Java, JavaScript, and Python. Based on the 16B
CodeGen-Multi model, CodeGen-Mono (16B) was obtained by further tuning on a
set of Python code collected from GitHub. Given a base model that was pre-
trained on 1 trillion tokens from the Stack dataset, the 15B StarCoder model
was obtained by training it on 35B tokens of Python code. WizardCoder further
empowers StarCoder with instruction tuning, following a similar instruction
evolution strategy as in WizardLM (Xu et al., 2023). CodeGeeX2, the second
generation of a multilingual generative model for code, is implemented based
on the ChatGLM2 architecture and trained on more code data. GPT-3.5-turbo is a
very capable commercial LLM developed by OpenAI and we accessed it in August,
2023. For these large LLMs, we tested pass@1 of all models except
GPT-3.5-turbo (whose result can be directly collected from Liu et al.
(2023a)’s paper). By sorting their pass@1 from high to low, they are ranked
as: GPT-3.5-turbo (61.7%), WizardCoder (46.23%, 15B), CodeGeeX2 (29.97%, 6B),
StarCoder (27.9%, 15B), CodeGen-Mono (26.15%, 16B), CodeGen2 (19.33%, 16B),
CodeGen-Multi (15.35%, 16B). The ranks on the MBPP dataset are similar.

Figure 1: Generating test cases for (a) self-generated code, (b) placeholder,
(c), oracle code, and (d) all-generated code.

##  4 Code to be Tested

For evaluating the testing ability of LLMs, we need an oracle to express the
ground-truth functionality of the tested code. Fortunately, current datasets
for evaluating program synthesis performance often provide such oracles (see
HuamnEval (Chen et al., 2021) and MBPP (Austin et al., 2021)). In our
experiments, we utilize an amended version of HumanEval called HumanEval+ (Liu
et al., 2023a), together with MBPP (the sanitized version). These datasets are
established to evaluate basic Python programming performance of LLMs, and they
contain 164 and 427 problems, respectively.

###  4.1 Imperfect Code Implementations

In order to simulate real-world scenarios where the tested code is often
buggy, we first adopt synthesized programs/code as the programs/code to be
tested, considering that the synthesis of even state-of-the-art LLMs is still
imperfect. We evaluate the performance of each LLM in testing code that was
generated by itself (which is denoted as “Self-generated”) and code in a set
consisting of program completion results of several different LLMs (which is
denoted by “All-generated”). That said, the compared LLMs take different code
implementations when generating test cases for each programming problem in the
self-generated setting. Whereas, in the all-generated setting, the same
program/code implementations are given to different LLMs for generating test
cases for comparison. In practice, we apply InCoder (1.3B), CodeGen2 (1B),
CodeT5+ (770M), and SantaCoder (1.1B) to construct the all-generated
program/code set, while, in the self-generated setting, each LLM first
synthesize code and complete a program to fulfill the requirement of each
programming problem, and the LLM then generates test cases with the
synthesized programs/code in its prompts. The temperature for all LLMs is
uniformly set to 0.2 for synthesizing the programs/code in both settings. We
obtain 100 program/code completions for each problem and we prompt each LLM to
generate 3 test cases for every program/code implementation in the self-
generated setting, and we sampled 100 implementations from the synthesis
results of InCoder (1.3B), CodeGen2 (1B), CodeT5+ (770M), and SantaCoder
(1.1B) to form the all-generated code set, i.e., we have $N=100$ for these
settings.

We follow the same way of generating code as introduced in the papers of these
LLMs. For model without instruction tuning, like InCoder and CodeT5+, we
synthesize programs/code using the default prompt given by each programming
problem in the test dataset, while, for models that have adopted instruction
tuning, e.g., WizardCoder, we use the recommended prompt in their papers.

Model | Size | Pass@1 | Pass@10 | Pass@100
---|---|---|---|---
InCoder | 1.3B | 6.99%/14.06% | 14.20%/34.98% | 23.76%/55.34%
CodeGen2 | 1B | 9.19%/17.50% | 16.06%/36.86% | 25.90%/59.32%
CodeT5+ | 770M | 12.95%/28.02% | 25.09%/47.69% | 37.56%/65.26%
SantaCoder | 1.1B | 15.21%/29.42% | 26.01%/51.30% | 43.80%/69.10%
Table 1: _Program synthesis performance_ of the _small_ LLMs (whose number of
parameters is around 1 billion) evaluated on HumanEval+ / MBPP (sanitized).

###  4.2 Optimal Code Implementations

As a reference, we also report the performance of generating accurate and
diverse test cases when the written code is perfectly correct, which is
achieved by adopting the oracle as the programs/code to be tested (and such a
setting is denoted by “Oracle”). Since Liu et al. (2023a) have reported that
some oracle code in the HumanEval dataset can be incorrect, we adopt the
amended oracle set in HumanEval+ in this setting. We further used the revised
oracle code implementations instead of the original ones in evaluating the
pass rate (i.e., $P^{\prime}$) of the generated test cases. Considering that
the public datasets often only provide one oracle implementation for each
problem, and to keep the uncertainty of evaluation results consistent, we copy
the oracle implementation by 100$\times$ and we prompt to generate 3 test
cases for each of these copies. It can be regarded as letting $N=100$, just
like in the previous settings in Section 4.1.

###  4.3 No Implementation

In certain scenarios, we require test cases before the function/program has
been fully implemented, hence we also evaluate in a setting where the main
body of a tested function/program is merely a placeholder, as depicted in
Figure 1(b). This scenario often occurs when the main code has not yet been
implemented for a function/program or the test engineer does not want to
introduce implementation bias to the LLM when generating test cases for a
function/program. We denote such a setting as “Placeholder” in this paper. We
also let $N=100$, as in the oracle setting.

##  5 Test Case Generation

In this section, we introduce how test cases can be generated, when the
implementation of a function/program is given as described in Section 4. In
this paper, a desired test case is a pair of input and its expected output for
the function/program defined in the context. As an example, Figure 1
demonstrates some test cases for the programming problem of checking whether
the two words satisfy a specific rotation pattern. To generate test cases, we
use the LLMs introduced in Section 3.

We wrote extra prompts to instruct the LLMs to generate three test cases for
each given code which include docstrings that describe the purpose of this
function, as depicted in Figure 1. Our instruction commands the LLMs (1) to
“check the correctness of this function with three test” and (2) to start
writing test code with an “assert” statement and the tested function, which
specifies the format of the test cases as input-output pairs that can be
parsed. For instance, given the example in Figure 1, the extra prompt should
be “# Check the correctness of this function with three test cases \n assert
cycpattern_check”.

We then concatenate the extra prompt with the code and feed the concatenation
into each LLM, for extracting test cases from the model output. The LLM will
try to complete the given input by generating one or more “assert”
statement(s), and we split the generation results into sub-strings, with
“assert” as the separator. Each sub-string is then considered as a test
statement, and we only take the first three statements if there exist more
than three statements, as has been introduced in Section 2. Such a split can
be considered as an effective post-processing operation which largely improves
the quality of the generated test code, considering that some non-sense code
pieces may be generated in the output of the LLMs. When using HumanEval+ and
MBPP, we try removing test cases in the docstrings of the function, if there
exist any, just to get rid of the broad hints from the docstrings (Chen et
al., 2023). The temperature for generating test cases is kept as 0.2.

Once obtained, the generated test cases are then compiled, and evaluated for
their correctness and diversity to report the pass rate $P^{\prime}$ and the
coverage rate $C$. When calculating, for each problem and every set of
completions generated, we create a temporary folder.

##  6 Main Results for Test Case Generation

Model | Size | Oracle | Self-generated | All-generated | Placeholder
---|---|---|---|---|---
InCoder | 1.3B | 21.31% (61.43%) | 23.37% (59.36%) | 22.72% (61.10%) | 25.19% (62.75%)
CodeGen2 | 1B | 31.63% (71.55%) | 30.62% (69.38%) | 30.93% (69.70%) | 30.69% (69.00%)
CodeT5+ | 770M |  35.43% (71.45%) |  32.34% (70.45%) |  31.49% (69.75%) |  32.67% (70.67%)
SantaCoder | 1.1B | 30.97% (71.46%) | 30.43% (70.81%) | 30.13% (70.55%) | 30.78% (71.24%s)
Table 2: The pass rates (and coverage rate) of the test cases generated on HumanEval+ in different settings for LLMs with around 1 billion parameters. Model | Size | Oracle | Self-generated | All-generated | Placeholder
---|---|---|---|---|---
CodeGen-Multi | 16B | 43.88% (67.91%) | 41.85% (69.30%) | 40.38% (66.97%) | 39.74% (68.28%)
CodeGen2 | 16B | 46.34% (73.07%) | 45.44% (73.17%) | 42.00% (72.45%) | 42.69% (72.86%)
CodeGen-Mono | 16B | 49.03% (74.82%) | 45.73% (73.74%) | 43.91% (73.66%) | 44.92% (73.63%)
StarCoder | 15B | 55.07% (76.02%) | 52.52% (72.45%) | 48.20% (72.30%) | 50.58% (74.52%)
CodeGeeX2 | 6B | 57.03% (74.42%) | 53.16% (73.55%) | 49.28% (70.32%) | 51.78% (73.08%)
WizardCoder | 15B | 53.89% (77.87%) | 55.47% (76.07%) | 48.02% (75.27%) | 49.89% (75.12%)
GPT-3.5-turbo | - |  71.03% (77.85%) |  72.45% (77.24%) |  59.24% (74.99%) |  66.28% (74.03%)
Table 3: The pass rates (and coverage rate) of the test cases generated on
HumanEval+ in different settings for LLMs whose parameters are obviously more
than 1 billion.

The experiment results of small and large LLMs on HumanEval+ can be found in
Table 2 and Table 3, respectively. Table 4 shows the results on MBPP. There
are several takeaways from these tables.

Figure 2: The correlation between code past rate and test pass rate in the
“Oracle” setting.

Figure 3: How the correctness of the test cases changes with their order when
being generated.

  * •

First, the test cases generated by LLMs can show a descent pass rate, and this
pass rate is even higher than the code pass rate on HumanEval+, which holds
for both large and small LLMs. Such a result is consistent with intuitions
from previous work which rejects code that cannot pass the generated tests to
improve the quality of program synthesis.

  * •

Second, the correctness of the generated test cases is positively correlated
with the LLM’s ability of generating code (see Figure 3, where each red cross
represents the performance of a model), which means an LLM showing the state-
of-the-art program synthesis performance is possibly also the state-of-the-art
LLM for program testing. As shown in Tables 2 and 3, GPT-3.5-turbo, which
synthesizes programs/code with the highest correctness, provides test cases
with the highest pass rate (71.03%) on HumanEval+. For an LLM, the more
accurate it is capable of synthesizing programs/code on a dataset, the more
powerful testing ability will probably be exhibited on the same dataset. There
also exist a few exceptions, e.g., SantaCoder (1.1B) outperforms CodeT5+
(770M) and CodeGen2 (1B) in generating code, but it shows inferior performance
in program testing on HumanEval+. By carefully examining the test cases
yielded by SantaCoder on HumanEval+, we found that it tends to generate more
complex and longer test cases than CodeT5+ for several problems on HumanEval+,
which are often more desirable in program testing. This is also why the
SantaCoder test cases show higher coverage rates in Table 2. To be concrete,
in Problem 131 in HumanEval+, where the program is required to return the
product of all digits with an odd position in a positive integer $n$ (which is
the input), the test input provided by CodeT5+ tends to be small for this
problem, e.g., $n=2$, while the SantaCoder test cases tend to have more digits
(e.g., $n=12358$), which is helpful in digging out hidden bugs. Yet,
generating longer and more complex test cases is more challenging, and the
correctness can be lower.

  * •

Third, as can be seen in Tables 3 and 4, generating test cases using _large_
LLMs with their self-generated code (in the prompts) often leads to a higher
level of correctness, compared with the placeholder results. This observation
is in fact unsurprising, considering that generating code first and test case
afterwards resembles the chain-of-thought prompting (Wei et al., 2022) (if
adopting the placeholder is regarded as a plain prompting), which is
beneficial to reasoning. Moreover, the self-generated performance of an LLM
sometimes even outperforms its testing performance with an oracle, and we
ascribe this to: 1) randomness in the style of the oracles which are few in
number and/or 2) less distribution shift between self-generated code in prompt
and the training code, for some powerful LLMs.

  * •

Fourth, with only a few exception, test cases obtained using the oracle code
exhibit slightly higher code coverage, while the coverage rate achieved in the
other settings (i.e., the self-generated, all-generated, and the placeholder
settings) is often slightly lower.

Model | Size | Oracle | Self-generated | All-generated | Placeholder
---|---|---|---|---|---
InCoder | 1.3B | 21.56% (46.81%) | 17.98% (46.11%) | 19.53% (46.45%) | 22.58% (46.72%)
CodeGen2 | 1B | 25.61% (54.26%) | 21.85% (53.09%) | 23.15% (50.43%) | 22.81% (52.11%)
CodeT5+ | 770M | 29.02% (56.86%) | 24.44% (52.31%) | 24.84% (53.20%) | 25.59% (55.81%)
SantaCoder | 1.1B |  32.37% (55.68%) |  26.40% (52.38%) |  26.20% (52.83%) |  26.53% (53.86%)
CodeGen-Multi | 16B | 41.32% (60.63%) | 35.96% (59.03%) | 34.17%,(58.09%) | 34.84% (58.92%)
CodeGen2 | 16B | 45.30% (62.15%) | 38.67% (60.16%) | 36.77% (58.59%) | 37.27% (59.16%)
CodeGen-Mono | 16B | 50.24% (64.39%) | 43.94% (62.94%) | 39.55% (61.99%) | 42.41% (62.31%)
StarCoder | 15B | 54.84% (65.10%) | 46.77% (63.60%) | 42.80% (61.95%) | 45.35% (62.66%)
CodeGeeX2 | 6B | 52.45% (64.64%) | 44.52% (63.72%) | 41.72% (60.48%) | 43.86%,(63.51%)
WizardCoder | 15B | 57.85% (66.68%) | 46.56% (64.86%) | 41.62% (60.72%) | 47.45% (64.54%)
GPT-3.5-turbo | - |  74.30% (66.19%) |  66.14% (65.30%) |  49.56% (62.95%) |  63.34% (64.72%)
Table 4: The pass rates (and coverage rate) of the test cases generated on
MBPP.

The above four takeaway messages can all be inferred from Tables 2, 3, and 4.
In addition to all these results, we conduct more experiments to achieve the
following takeaway messages.

  * •

Fifth, by analyzing the relationship between the quality of code in prompts
and the correctness of test, we found that correct code implementation in the
prompt often leads to higher quality of test code generation than the case
when some incorrect code is given. We conducted an experiments where we first
select programming problems in HumanEval+, where the code pass rate of an LLM
is neither 0% or 100%. Then we separate self-generated programs/code of the
model into two groups, with one group only contains programs/code that are
considered as correct and the other only contains incorrect programs/code. In
Table 5, we compare the performance of using these two sorts of code in the
prompt, for generating test cases using the same LLM. Apparently, the quality
of test cases obtained with correct programs/code is obviously higher. We
further evaluate the overall testing performance of LLMs with only correct
self-generated programs/code, if there exists any, in their prompts. Unlike in
Table 5 where we do not take problems that can be 100% or 0% solved, we take
all given problems in this evaluation, except, for every problem, we eliminate
all incorrect self-generated programs/code if there exist at least one correct
implementation synthesized by the evaluated LLM. By doing so, we can observe
substantially improved program testing ability on HumanEval+ (i.e., 74.95% for
GPT-3.5-turbo, 56.87% for WizardCoder, 54.33% for CodeGeeX2, and 53.24% for
StarCoder), comparing with the original self-generated results in Table 3. The
same on MBPP. Recall that, in our third takeaway message, we have mentioned
that test cases obtained with self-generated programs/code sometimes even
outperform those yielded in the oracle setting on HumanEval+, maybe partly due
to less distribution shift between the synthesized programs/code and the
training code, the above results further confirm that, if we can improve the
correctness of synthesized programs/code while keeping proper styles, then
more powerful testing ability can further be achieved.

  * •

Sixth, by conducting an additional experiment, we further compare the quality
of test cases collected from different positions in the generation results.
For every set of the three generated test cases, we analyze the relationship
between their correctness and the order when they are generated. The results
are illustrated in Figure 3. As can be seen in the figure, the first generated
test case often shows the best correctness and the latterly generated ones are
more incorrect. This may be due to the fact that the model tends to first
generate content with a high level of confidence (which is also more likely to
be correct).

##  7 Improving Program Synthesis Using the Generated Test Cases

High quality test cases are not only desired in program analyses, but also
helpful to program synthesis. Previous methods have successfully used
generated test cases to improve the performance of LLMs in synthesizing
programs/code. For instance, Li et al. (2023a) designed a special prompt which
involves the test cases as an preliminary, if they are available, for
generating programs/code. Shi et al. (2022) introduced a Bayes risk decoding
mechanism, which executes generated code from a candidate set on a small
number of test inputs and selects by marginalizing over implementations that
share the same outputs when given these test inputs. It utilizes consistency
between the output of code that is correctly implemented. One step further,
Chen et al. (2023) proposed CodeT, which leverages the LLM to obtain test
cases first and tests all synthesized programs/code with these test cases by
performing a dual execution agreement, which considers both the agreement
between the execution output and the test output and the consistency between
the output of correct program implementations, to obtain state-of-the-arts. We
encourage interested reader to read the original paper.

Model | Size | w/ correct code | w/ incorrect code | #Problem
---|---|---|---|---
InCoder | 1.3B | 28.55% | 27.39% | 27
CodeGen2 | 1B | 27.25% | 25.74% | 11
CodeT5+ | 770M | 40.19% | 36.78% | 27
SantaCoder | 1.1B | 37.45% | 34.08% | 24
CodeGen-Multi | 16B | 55.49% | 50.06% | 32
CodeGen2 | 16B | 43.56% | 39.31% | 29
CodeGen-Mono | 16B | 45.18% | 42.86% | 56
StarCoder | 15B | 58.16% | 57.08% | 68
CodeGeeX2 | 6B | 52.84% | 48.63% | 51
WizardCoder | 15B | 48.02% | 45.12% | 54
GPT-3.5-turbo | - | 75.39% | 68.52% | 126
Table 5: With the correct (self-generated) code, the LLMs show stronger
ability of generating correct test cases on HumanEval+ (evluated only on those
problems that can neither be 0% solved nor 100% solved), than in the case
where incorrect self-generated code is given in the prompts. Since most LLMs
cannot generate any correct code for many hard problems while they often
generate incorrect code even for easy problems, the number of tested problems
in this experiment increases with the power of the tested LLM, as shown in the
rightmost column.

In the previous section, we have obtained results about many intriguing
properties of the program testing performance of LLMs for code. In this
section, we would like to drive the readers to think whether it is possible to
utilize these results to improve the program synthesis performance,
considering that the test cases (hand-crafted and given or automatically
generated in particular) are widely and successfully used in program
synthesis. We shall demonstrate that, by utilizing takeaway messages in
Section 6, the program synthesis performance of previous methods can be
improved significantly. Taking CodeT as an example of the previous state-of-
the-art, the method uses a placeholder to generate test cases and treats all
the test cases as equally correct as a prior. However, as discussed in our
third takeaway message, using self-generated code helps to achieve more
powerful ability in generating correct test cases. Moreover, if multiple test
cases are provided in a single run of generation given an LLM, the correctness
of the test cases decreases with their generation order, as shown in our fifth
point. Hence, to obtain superior program synthesis performance, we introduce
two simple modifications to it: 1) we employ the “self-generated” setting
instead of the “placeholder” setting for generating test cases, which means we
synthesize programs first and then generate test cases for each program in the
prompt, 2) we assign different weights to the generated test cases based on
their order in each generation result.

We test the effectiveness of using 1) the prompt which involves self-generated
(SG) code and 2) the rank-based weighted (RW) test cases, in improving program
synthesis performance on HumanEval+. The details of our implementation are
introduced as follows. Following Chen et al. (2023), we used a temperature of
0.8 to generate code and self-generated test cases. Each test case is weighted
by $p^{i-1}$ with $i$ being its order in the model output, and we let $p=0.8$.

Table 6 shows the results. We compare CodeT with CodeT+SG, CodeT+RW, and
CodeT+SG+RW. For CodeT, we follow their official implementation and generate
$100\times 5$ test cases for each problem. For fair comparison, we ensure that
our solutions with SR and/or RW generate the same numbers of program
implementations and test cases as CodeT does. Hence, for each problem in
HumanEval+, we synthesize a program together with its $5$ test cases for $100$
times when SR and/or RW are incorporated, i.e., we have $i\in\\{1,2,3,4,5\\}$.
It can be seen from the table that both SG and WR improves the program
synthesis performance considerably on most LLMs, except for Incoder,
CodeGen2-1B, CodeT5+, and SantaCoder for which the test cases generated in the
placeholder setting show similar or even higher correctness than in the self-
generated setting and SG fails with them. For some LLMs, SG is more powerful,
while, on the other models including SantaCoder and StarCoder, RW is more
powerful. By combining SG and RW, the program synthesis performance of most
powerful LLMs in Table 6 improves, comparing to only using one of the two. On
GPT-3.5-turbo and WizardCoder, which are the best two models in synthesizing
programs on HumanEval+, we achieve +4.22% and +3.04% performance gains for
CodeT, respectively, with SG & RW.

We believe there exist other ways of using our takeaway messages for improving
the program synthesis performance, and we would like to encourage future work
to explore more in this direction.

Model | Size | Baseline | CodeT | \+ SG | \+ RW | \+ SG & RW
---|---|---|---|---|---|---
InCoder | 1.3B | 6.99% | 9.85% | 9.45% | 10.26% | 9.98%
CodeGen2 | 1B | 9.19% | 15.15% | 14.89% | 15.67% | 15.35%
CodeT5+ | 770M | 12.95% | 16.57% | 16.28% | 17.19% | 16.98%
SantaCoder | 1.1B | 15.21% | 18.43% | 18.17% | 18.75% | 18.63%
CodeGen-Multi | 16B | 15.35% | 24.50% | 25.71% | 25.72% | 26.95%
CodeGen2 | 16B | 19.33% | 27.56% | 28.51% | 28.43% | 29.63%
CodeGen-Mono | 16B | 26.15% | 35.63% | 36.69% | 36.63% | 37.95%
StarCoder | 15B | 27.90% | 40.46% | 41.21% | 42.12% | 43.15%
CodeGeeX2 | 6B | 29.97% | 44.16% | 45.23% | 44.92% | 46.32%
WizardCoder | 15B | 46.23% | 58.41% | 60.13% | 59.60% | 61.45%
GPT-3.5-turbo | - | 61.70% | 69.25% | 72.45% | 70.75% | 73.47%
Table 6: Program synthesis performance (Pass@1) of LLMs can be significantly
improved by using our takeaway messages in Section 6. The experiment is on
HumanEval+.

##  8 Related Work

Test case generation via program analysis.  Generating reasonable test cases
for analyzing programs is a long standing problem in the software engineering
community. Various program analysis techniques, e.g., fuzzing, have been
developed for achieving this goal. AFL++ (Fioraldi et al., 2020) is the most
popular tool which incorporate many techniques in this category. A major
weakness of these techniques is understandability of the generated test cases.

Test case generation via deep learning.  The invention of transformer (Vaswani
et al., 2017) and self-supervised pre-training (Devlin et al., 2018; Lewis et
al., 2019; Raffel et al., 2020; Radford et al., 2018) have brought a
breakthrough to programming language processing. After being trained in a
self-supervised manner on a large and diverse code corpus, LLMs have
demonstrated remarkable abilities in understanding and synthesizing programs.
We have also witnessed the adaptation of pre-trained LLMs (e.g., ChatGPT) to
fuzz testing (Xia et al., 2023) very recently. Nevertheless, there still lack
and requrie in-depth analyses and intensive comparisons of different LLMs in
program testing. In particular, powerful LLMs emerge continuously. For
instance, the recent WizardCoder (Luo et al., 2023) exhibits an obivous
program synthesis superiority over other open-source LLMs. In our study, we
focus on the analyses and comparison of the LLMs in writing test code and
generating test cases.

Evaluation of Large Language Model.  Recently, large language models (LLMs)
has incited substantial interest in both academia and industry. In order to
evaluate the capabilities of large language models, a variety of effort have
been devoted from the perspectives of natural/programming language processing
accuracy, robustness, ethics, biases, and trustworthiness, etc. For instance,
PromptBench (Zhu et al., 2023) demonstrates that current LLMs are sensitive to
adversarial prompts, and careful prompt engineering is necessary for achieving
descent performance with them. Another example, DecodingTrust (Wang et al.,
2023a), offers a multifaceted exploration of trustworthiness of the GPT
models, especially GPT-3.5 and GPT-4. The evaluation expands beyond the
typical trustworthiness concerns to include several new critical aspects.
Agentbench (Liu et al., 2023b) evaluates LLM as agents on challenging tasks in
interactive environments. Their experimental results show that, while top
commercial LLMs present a strong ability of acting as agents in complex
environments, there is a significant disparity in performance between them and
their open-source competitors.

##  9 Conclusion

In this paper, we have performed thorough analyses of recent LLMs (mostly LLMs
for code) in testing programs/code. Through comprehensive experiments with 11
LLMs on programming benchmark datasets including HumanEval+ and MBPP (the
sanitized version), we have uncovered a range of intriguing characteristics of
these LLMs for program/code testing. We have illustrated how the program
testing capabilities of these LLMs can be enhanced in comparing intensive
empirical results in four different settings. Based on our findings, we are
also capable of improving the performance of state-of-the-art LLMs in
synthesizing programs/code with test cases of higher quality. As a preliminary
research work, we believe our paper can provide new research insights and
spark new ideas in program/code synthesis, test-case generation, and LLM
understanding, and we look forward to future exploration in this direction in
future work.

## References

  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ , 2023.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Chen et al. (2023) Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen.  Codet: Code generation with generated tests.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=ktrw68Cmu9c.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fioraldi et al. (2020) Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse.  $\\{$AFL++$\\}$: Combining incremental steps of fuzzing research.  In _14th USENIX Workshop on Offensive Technologies (WOOT 20)_ , 2020\.
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=hQwb-lbM6EL.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Kocetkov et al. (2023) Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries.  The stack: 3 TB of permissively licensed source code.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=pxpbTdUEpD.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. (2023a) Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin.  Towards enhancing in-context learning for code generation.  _arXiv preprint arXiv:2303.17780_ , 2023a.
  * Li et al. (2023b) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ , 2023b.
  * Lin (2004) Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Liu et al. (2023a) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _arXiv preprint arXiv:2305.01210_ , 2023a.
  * Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.  Agentbench: Evaluating llms as agents, 2023b.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023.
  * Nijkamp et al. (2023a) Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou.  Codegen2: Lessons for training llms on programming and natural languages.  _arXiv preprint arXiv:2305.02309_ , 2023a.
  * Nijkamp et al. (2023b) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis, 2023b.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Shi et al. (2022) Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang.  Natural language to code translation with execution.  _arXiv preprint arXiv:2204.11454_ , 2022.
  * Tufano et al. (2020) Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan.  Unit test case generation with transformers and focal context.  _arXiv preprint arXiv:2009.05617_ , 2020.
  * Tufano et al. (2022) Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy.  Methods2test: A dataset of focal methods mapped to test cases.  In _Proceedings of the 19th International Conference on Mining Software Repositories_ , pp. 299–303, 2022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2023a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, 2023a.
  * Wang et al. (2023b) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_ , 2023b.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Xia et al. (2023) Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang.  Universal fuzzing via large language models.  _arXiv preprint arXiv:2308.04748_ , 2023.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ , 2023.
  * Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.  _arXiv preprint arXiv:2303.17568_ , 2023.
  * Zhu et al. (2023) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie.  Promptbench: Towards evaluating the robustness of large language models on adversarial prompts, 2023.

##  Appendix A Appendix

###  A.1 Further Analysis of Experimental Results

In this part, we provide further analysis of the experimental results in
Section 6.

With regard to the situation where the test case quality generated by
SantaCoder is lower than that generated by CodeT5+ on the HumanEval+ dataset,
we have explained that this is probably because SantaCoder tends to generate
longer and more complex test cases. Here we further demonstrate that
SantaCoder is capable to generate more accuracy output when given the same
testing input as that of CodeT5+’s. To show this, we first extract the input
part of the test cases (which includes testing inputs paired with their
corresponding outputs) generated by CodeT5+ in the oracle setting. We then let
SantaCoder to generate testing outputs given these inputs, and assessed the
accuracy of such test cases. The results show that, given these testing inputs
already, SantaCoder and CodeT5+ obtain an correctness of 41.67% and 40.34%,
respectively, showing that SantaCoder is indeed stronger, if the same testing
input is given and it does not have the chance to yeild more complex testing
inputs.


# Weakly Supervised Open-Vocabulary Object Detection

Jianghang Lin1, Yunhang Shen2, Bingquan Wang1, Shaohui Lin3, Ke Li2, Liujuan
Cao1 Corresponding Author

###### Abstract

Despite weakly supervised object detection (WSOD) being a promising step
toward evading strong instance-level annotations, its capability is confined
to closed-set categories within a single training dataset. In this paper, we
propose a novel weakly supervised open-vocabulary object detection framework,
namely WSOVOD111Source code are available at: https://github.com/HunterJ-
Lin/WSOVOD, to extend traditional WSOD to detect novel concepts and utilize
diverse datasets with only image-level annotations. To achieve this, we
explore three vital strategies, including dataset-level feature adaptation,
image-level salient object localization, and region-level vision-language
alignment. First, we perform data-aware feature extraction to produce an
input-conditional coefficient, which is leveraged into dataset attribute
prototypes to identify dataset bias and help achieve cross-dataset
generalization. Second, a customized location-oriented weakly supervised
region proposal network is proposed to utilize high-level semantic layouts
from the category-agnostic segment anything model to distinguish object
boundaries. Lastly, we introduce a proposal-concept synchronized multiple-
instance network, i.e., object mining and refinement with visual-semantic
alignment, to discover objects matched to the text embeddings of concepts.
Extensive experiments on Pascal VOC and MS COCO demonstrate that the proposed
WSOVOD achieves new state-of-the-art compared with previous WSOD methods in
both close-set object localization and detection tasks. Meanwhile, WSOVOD
enables cross-dataset and open-vocabulary learning to achieve on-par or even
better performance than well-established fully-supervised open-vocabulary
object detection (FSOVOD).

(a) Paradigm comparisons.

(b) Qualitative comparisons.

Figure 1:  Vanilla WSOD is confined to detecting known categories from the
training set, _e.g._ , Person, and Horse. The proposed WSOVOD is generalized
to unseen categories, _e.g._ , Flag, and Fence. WSOVOD outperforms the
previous state-of-the-art WSOD methods and achieves on-par or even better
performance than FSOVOD.

## Introduction

In the past decades, the artificial intelligence community has witnessed great
progress in object detection. In particular, large amount of human-annotated
datasets significantly promotes the prosperity and progress of fully-
supervised object detection (FSOD), such as Faster RCNN (Ren et al. 2015),
YOLO (Redmon et al. 2016; Redmon and Farhadi 2017), Detr (Carion et al. 2020)
and their variants (Zhu et al. 2021; Zheng et al. 2021; Sun et al. 2021).
Nevertheless, the laborious and lavish collection of instance-level
annotations has severely barricaded the applicability of FSOD in practical
application with large-scale categories. By excavating image-level category
supervision that indicates the presence or absence of an object, weakly
supervised object detection (WSOD) has attracted much attention recently since
image-level annotations are widely available in easily-collected
classification-like datasets.

Unfortunately, a de facto limitation of the existing WSOD methods (Tang et al.
2017, 2018; Kim et al. 2020) stems from their detectors only concentrating on
few categories concepts within individual datasets, such as $20$-category
Pascal VOC (Everingham et al. 2010) and $80$-category MS COCO (Lin et al.
2014). Little effort has been made to explore the limit of WSOD learning at
scale toward detecting novel objects. Thus, it may not fully exploit the
latent capacity of WSOD whose original intention is to leverage the tremendous
amount of tagged images to train object detectors. To solve the above
limitation, as illustrated in Fig. 1 (a), we extend WSOD settings to detect
and localize open-vocabulary concepts using joint large-scale weakly-annotated
datasets that are publicly available. Accordingly, a weakly supervised open-
vocabulary object detection, referred to as WSOVOD is put forth.

To this end, three main challenges, as we start in this paper, obstacle to the
implementation of WSOVOD. First, non-identical data distributions may bring
dataset bias (Kim, Lee, and Choo 2021; Torralba and Efros 2011; Jiang et al.
2022) to affect the feature learning, hindering the vision-language alignment
introduced as followed. For example, ILSVRC (Russakovsky et al. 2015) is an
object-centric dataset with a balanced category distribution, while LVIS
(Gupta, Dollár, and Girshick 2019) has many complex scenes with Zipfan
distribution. Second, the reliance of existing WSOD methods upon traditional
object proposal generators prevents models from learning proposal extraction
at different semantic levels, since they only use low-level features computed
on super-pixel (Felzenszwalb and Huttenlocher 2004) or counters (Dollár and
Zitnick 2013). Though UWSOD (Shen et al. 2020a) learns object proposal
generator under pseudo-label supervision, it is difficult to generate high-
quality proposals due to noisy training. Third, weak supervision hardly aligns
vision-language representation. In the existing open-vocabulary studies (Ma et
al. 2022a; Gu et al. 2022; Zang et al. 2022), the visual-semantic alignments
are realized in a fully-supervised manner where classification embeddings and
box knowledge are necessary. Though recent methods (Zhou et al. 2022b; Kamath
et al. 2021; Zareian et al. 2021) resort to weak information, e.g., captions,
they deeply rely on strong box-level annotations.

To solve the above three problems and overcome the limitations of common WSOD
approaches, our WSOVOD framework (in Fig. 2) innovates in three aspects: 1) We
extract data-aware features to generate for each image input-conditional
coefficients and combine dataset attribute prototypes to identify dataset bias
in proposal features of different distributions. Explicitly, an additional
branch learns to squeeze the global image feature into a channel-wise global
vector as coefficients to weight dataset attribute prototypes for re-
calibrating final proposal features, which effectively helps achieve model
generalization to different scenarios and categories. 2) A location-oriented
region proposal network is proposed to leverage high-level semantic layouts
from the image segmenter to distinguish object boundaries. Recent interactive
segmentation work SAM (Kirillov et al. 2023) exhibits strong image
segmentation capabilities, but it lacks semantic recognition ability. Here, we
transfer the knowledge from SAM to a customized region proposal network upon
high-quality proposals. 3) We introduce a proposal-concept synchronized
multiple-instance network that implements object mining to discover objects
under image-level classification embeddings, as well as instance refinement to
align vision-language representation. Specifically, we obtain text embeddings
of the target vocabulary from the pre-trained text encoder, which are
considered as category prototypes for multiple-instance learning. Also, we
transform the multi-branch refinement heads in the common WSOD framework into
open-vocabulary learning to further align object and concept representation.
In addition, we leverage SAM to refine the box coordinates of the supervision
between multi-branch refinement heads.

Extensive experiments on Pascal VOC and MS COCO demonstrate that the proposed
WSOVOD achieves on-par or even better performance compared to fully-supervised
open-vocabulary detection methods, which paves a new way to explore the large
number of visual concepts from image-level supervisions. For example, our
method significantly outperforms OVR-CNN (Zareian et al. 2021), ViLD (Gu et
al. 2022) and Detic (Zhou et al. 2022b) that require box-level annotations of
base categories, by $13.9\%$, $9.1\%$ and $8.9\%$ AP, respectively, for novel
categories in MS COCO. Moreover, WSOVOD achieves new state-of-the-art
performance compared to the previous WSOD methods under the close-set and
single-dataset settings while being able to detect novel categories.

## Related Work

### Weakly Supervised Object Detection

Combining multiple-instance learning (MIL) (Dietterich, Lathrop, and Lozano-
Pérez 1997) with convolutional neural networks (CNNs) has made great progress
in WSOD. WSDDN (Bilen and Vedaldi 2016) is the prior work to introduce MIL
into CNN and model WSOD as a proposal classification. However, WSDDN suffers
from local optimization problems since the detector tends to detect high-
activated regions. OICR (Tang et al. 2017) further attaches multi-branch
refinement to WSDDN, which gradually propagates the scores of the salient
regions to the complete objects. These methods are highly dependent on
traditional proposal generation methods (Uijlings et al. 2013; Pont-Tuset et
al. 2016) and do not regress the final proposal boxes. Furthermore, UWSOD
(Shen et al. 2020a) learns multi-scale features and the region proposal
network in an end-to-end unified framework. Nevertheless, the region proposal
network is prone to be saturated due to the noisy pseudo-ground-truth boxes in
the early training period, which has inferior performance than the cutting-
edge WSOD methods. Different from these methods, we exploit knowledge transfer
from the category-agnostic segmenter to pursue high-quality and high-recall
object proposals.

Figure 2: Illustration of the proposed WSOVOD framework. The proposal
generator combines candidate regions from LO-WSRPN and SAM that may
potentially contain objects for subsequent object mining. The data-aware
feature extractor outputs unbiased dataset attribute features by identifying
dataset bias from dataset attribute prototypes. The proposal-concept
synchronized multiple-instance learning discovers potential objects that match
the target vocabularies in image-level labels.

### Open-Vocabulary Object Detection

Open-vocabulary object detection (OVOD) (Zareian et al. 2021; Gu et al. 2022;
Minderer et al. 2022; Zhou et al. 2022b) is an attractive research topic in
recent years, whose goal is to detect unseen or novel classes that occupy a
particular semantically-coherent region within an image. OVOD differs from
zero-shot object detection (Bansal et al. 2018) in that it can access large-
scale novel objects with weakly-supervised labels, e.g., tags, and captions.
However, they share the same paradigm of learning a cross-modal vision-
language representation space to model image regions and word descriptors. The
main challenge in this field is aligning proposal features with category text
embeddings, thus it is crucial to use image-text knowledge efficiently
(Radford et al. 2021; Li et al. 2022). OVR-CNN (Zareian et al. 2021) pre-
trains the detector on image-text pairs using contrastive learning and fine-
tunes it on detection data with a limited vocabulary. OWL (Minderer et al.
2022) further transfers knowledge from vision-language models to transformer-
based detectors with contrastive image-text pre-training and detection fine-
tuning. Detic (Zhou et al. 2022b) improves OVOD performance of long-tail
categories via image-level annotated data, such as ImageNet (Jia et al. 2009)
and conceptual captions (Sharma et al. 2018). Different from these approaches,
our proposed WSOVOD uses MIL-based object mining to discover potential objects
and refines them by multi-branch refinement open-vocabulary heads gradually.
All of the above methods are highly dependent on bounding-box annotations,
while WSOVOD is devoted to efficiently exploring weakly-annotated data.

## Methodology

As illustrated in Fig. 2, an image $\mathcal{I}$ first goes through the vision
backbone to extract global image features $X^{\mathrm{img}}$. Then, the data-
aware feature extractor takes in $X^{\mathrm{img}}$ to generate coefficients
for combining dataset attribute prototypes as data-aware features
$X^{\mathrm{daf}}$. Meanwhile, a proposal generator also takes in
$X^{\mathrm{img}}$ to hypothesize object locations. Next, RoI pooling crops
the pooled features from global feature $X^{\mathrm{img}}$, and two fully-
connected layers transform them to get proposal features
$X^{\mathrm{prop}}\in\mathcal{R}^{R\times D}$, where $R$ is proposal number in
image $\mathcal{I}$ and $D$ is feature vector length. We further fuse proposal
features $X^{\mathrm{prop}}$ with data-aware features $X^{\mathrm{daf}}$ to
deal with dataset bias, resulting in $X^{\mathrm{fuse}}$. Finally, a proposal-
concept synchronized multiple-instance learning takes in $X^{\mathrm{fuse}}$
to discover objects constrained by image-level classification embeddings and
align representation between objects and concepts. The overall training
objective function is formulated as:

$\mathcal{L}_{\mathrm{WSOVOD}}=\mathcal{L}_{\mathrm{PG}}+\mathcal{L}_{\mathrm{OM}}+\mathcal{L}_{\mathrm{IR}},$
(1)

where $\mathcal{L}_{\mathrm{PG}}$ is proposal generator loss. And
$\mathcal{L}_{\mathrm{OM}}$ and $\mathcal{L}_{\mathrm{IR}}$ are object mining
and instance refinement losses for proposal-concept synchronized multiple-
instance learning.

### Data-Aware Feature Extractor

To better align vision-language representation, it is necessary to learn as
many categories as possible, however, an individual dataset contains limited
concepts. This motivates us to train one detector upon multiple datasets
jointly to generalize the detection scope of WSOD. The main challenge stems
from domain incompatibility over non-identical data distribution. Much of the
bias can be accounted for by the divergent goals of the different datasets:
For example, LVIS (Gupta, Dollár, and Girshick 2019) has an average of $11.2$
instances from $3.4$ categories per image with long-tail Zipfan distribution,
while most images in ILSVRC (Russakovsky et al. 2015) are object-centric with
single category. Such variant dataset biases hurt the representation learning,
thus simply combining multiple datasets has poor performance as observed in
our experiments. In contrast, such bias could be well recognized even from a
single image by humans and classifiers (Torralba and Efros 2011).

To this end, we design a data-aware feature extractor (DAFE) to generate
generalized dataset-level features for cross-dataset learning with different
scenarios and different categories. The key intuition of DAFE is to capture
the unique and identifiable “signature” of each dataset conditioned on full-
image information and adjust proposal features accordingly. From an
alternative perspective, DAFE shifts the proposal position in the visual-
semantic embedding space via visual prompt, as text embeddings are fixed.
Specifically, it consists of a global average pooling layer to squeeze the
input information from image feature maps $X^{\mathrm{img}}$. Then two fully-
connected layers followed by the Tanh activation function learn to generate
coefficients based on the image input to combine dataset attribute prototypes
for identifying the dataset bias from the squeezed global features and produce
data-aware feature $X^{\mathrm{daf}}$ with the same dimension of the proposal
features. Finally, we aggregate $X^{\mathrm{daf}}$ with the proposal features
$X^{\mathrm{prop}}$ by element-wise summation:
$X^{\mathrm{fuse}}=X^{\mathrm{prop}}+X^{\mathrm{daf}}$. Thus, the input-
conditional vector $X^{\mathrm{daf}}$ aims to re-calibrate the original
proposal features to de-bias the different distributions, which are then used
for the subsequent open-vocabulary object mining and refinement. Albeit the
apparent simplicity, DAFE approach plays a vital role in cross-dataset
learning.

Discussion. The proposed DAFE, to some extent, is related to recent prompt
tuning (Jia et al. 2022) that adapts large foundation models to downstream
tasks with a small amount of task-specific learnable parameters. Our approach
differs from theirs in two folds. First, most prompt learning methods perform
data-space adaptation by transforming the input. For example, approaches in
(Feng et al. 2022) append a sequence of learnable vectors to the textual
input, and method in (Bahng et al. 2022) learns an image perturbation to
convert the image to the formats of downstream tasks. Different from the above
methods, our DAFE eliminates the different dataset distributions by feature-
space adaptation with an input-conditional vector. Second, existing prompt
tuning mainly focuses on fully-supervised learning, which is difficult to
generalize to wide unseen categories. Input-conditional prompt learning (Zhou
et al. 2022a) still relies on an online text encoder to generate input-
specific weights for each image. Our adaptation does not require an online
text encoder during training and inference.

### SAM Guided Proposal Generator

Most existing WSOD methods use traditional proposal methods with low-level
features to generate region candidates, which prevents the models from end-to-
end learning with high-level semantic information. We design a location-
oriented weakly supervised region proposal network (LO-WSRPN) to recognize
category-agnostic potential objects, which further transfers knowledge of
high-level semantic layouts from SAM (Kirillov et al. 2023). In detail,
similar to RPN from Faster RCNN (Ren et al. 2015), LO-WSRPN has a $3\times 3$
convolutional layer with $256$ channels followed by three sibling $1\times 1$
convolutional layers for localization and shape quality estimations,
respectively. The first two convolutional layers are responsible for
localization quality estimation, predicting centerness $c$ and foreground
probabilities $p$, respectively. We use $s=\sqrt{c\cdot p}$ as the
localization quality for each region proposal during inference. The last
convolutional layer is responsible for shape quality estimation. Different
from anchor-based detectors, we directly view locations as training samples
instead of anchor boxes. We replace the standard box-delta targets $(x,y,h,w)$
with distances $t=(l,r,t,b)$ from the location to four sides of the ground-
truth box as in (Tian et al. 2019). The training objective function of this
module is formulated as:

$\begin{split}\mathcal{L}_{\mathrm{PG}}=&\mathcal{L}_{\mathrm{BCE}}(p,p^{*})+\\\
&\mathds{1}_{p^{*}=1}\\{L_{1}(c,c^{*})+\mathcal{L}_{\mathrm{IoU}}(t,t^{*})\\},\end{split}$
(2)

where $\mathcal{L}_{\mathrm{BCE}}$ is the binary cross-entropy loss function,
$L_{1}$ constrains the distance between the sampling anchor points and the
pseudo-ground-truth (PGT) boxes, $\mathcal{L}_{\mathrm{IoU}}$ measures the
shape difference between the predicted boxes and the PGT boxes, thereby
constraining the shape of the predicted boxes. We use WSOVOD’s final
predictions as PGT boxes and assign the corresponding targets, i.e., $p^{*}$,
$c^{*}$, and $t^{*}$.

However, object proposals from LO-WSRPN are extremely noisy in the early stage
of training as observed in (Shen et al. 2020a), which has a negative impact on
subsequent object mining and brings in inferior PGT boxes. Inspired by large-
scale interactive segmentation models (Kirillov et al. 2023), we leverage SAM
to generate additional proposals during training, which helps stabilize
subsequent object mining. In detail, we first sprinkle evenly $32\times 32$
grid points as the prompt input of SAM to generate additional proposals. Then,
we concatenate the SAM proposals with the learned proposals from LO-WSRPN as
input to subsequent object mining. Incorporating knowledge from SAM not only
helps enrich the high-quality object proposals but also leverages high-level
semantic layouts from the image segmenter to distinguish object boundaries.

### Proposal-Concept Synchronized Multiple-Instance Network

The central idea of fully-supervised open-vocabulary object detection (FSOVOD)
is to align object features with text embeddings which are pre-trained on
large-scale image-text pairs like CLIP (Radford et al. 2021). In detail,
FSOVOD methods convert a generic two-stage object detector to an open-
vocabulary detector by replacing the learnable classifier head with fixed
language embeddings, corresponding to the category names. Thus, object-level
annotations are required during training to maximize the embedding
similarities of positive region-category pairs and minimize that of negative
ones. However, it is challenging to align object-level vision-language
representation with only image-level supervision. Fortunately, WSOD is often
formulated as multiple-instance learning (MIL) (Dietterich, Lathrop, and
Lozano-Pérez 1997) and implicitly learns instance-based classifier from image-
level information.

Therefore, our WSOVOD extends the common MIL-based WSOD framework (Bilen and
Vedaldi 2016) to mine large-scale category concepts in an open-vocabulary
manner. The fused proposal features $X^{\mathrm{fuse}}$ are forked into two
fully-connected layers parallel, namely classification stream
$W^{c}\in\mathcal{R}^{D\times C}$ and detection stream
$W^{d}\in\mathcal{R}^{D\times C}$, producing two score matrices
${S}^{c},{S}^{d}\in\mathcal{R}^{R\times C}$ respectively, where $C$ and $R$
denote the number of categories and proposals during training in image $I$,
respectively. Different to work in (Bilen and Vedaldi 2016), we adapt text
embedding $T\in\mathcal{R}^{D\times C}$ of category names as the parameters
$W^{c}$ of classification stream so that it imposes explicit visual-semantic
constraint during MIL optimization. At the same time, the detection stream is
still learnable, since it focuses on localizing the foreground proposals,
which is expected to be category-agnostic. Thus, the two score matrices are
computed as: $S^{c}=\frac{X^{\mathrm{fuse}}}{\lVert
X^{\mathrm{fuse}}\rVert}\frac{T}{\lVert T\rVert}$ and
$S^{d}=X^{\mathrm{fuse}}W^{d}$. Then, both score matrices are normalized by
softmax functions $\sigma(\cdot)$ over category and proposal axes,
respectively. The final score ${S}$ for assigning category $c$ to region $r$
is computed via an element-wise product:
${S}=\sigma({S}^{c})\odot\sigma(({S}^{d})^{T})^{T}\in[0,1]$. To acquire image-
level classification scores for training, $S$ is summed for all regions
$\varphi_{c}=\sum_{r=1}^{R}S_{rc}$. Then the object-mining objective function
is binary cross-entropy loss:

$\mathcal{L}_{\mathrm{OM}}=\sum_{c=1}^{C}{y}_{c}\log(\varphi_{c})+(1-{y}_{c})\log(1-\varphi_{c}),$
(3)

where ${y}\in\\{0,1\\}^{C}$ is the category one-hot label indicating image-
level existence of category $c$.

Recently, WSOD works (Tang et al. 2017, 2018) also explicitly assign pseudo
labels from the above mining module to learn more discriminative classifiers,
which are also called instance refinement modules. Thus, we also develop
multiple open-vocabulary classification heads which uses a shared vision-
language representation space to refine discovered object. In addition, to
reduce miss-localization, for each refinement branch we regress the bounding
boxes which need high-quality proposals to provide PGT boxes as the previous
WSOD methods (Yang, Li, and Dou 2019; Ren et al. 2020). Therefore, the PGT
boxes mined by the object mining module are used as box prompt input to SAM to
refine boxes to supervise the first refinement branch, and the former
refinement branch supervises the latter one. Thus, the objective function of
this multi-branch refinement is the sum of classification and regression
losses over all branches:

$\mathcal{L}_{\mathrm{IR}}=\sum_{k=1}^{K}\mathcal{L}_{\mathrm{cls}}^{k}+\mathcal{L}_{\mathrm{reg}}^{k},$
(4)

We concatenate the text embedding $T$ with a background zero-vector as the
classifier parameters $W^{r}\in\mathcal{R}^{D\times(C+1)}$ of refinement
branch $k$. The classification loss is defined as:

$\mathcal{L}_{\mathrm{cls}}^{k}=\sum_{r=1}^{R}\sum_{c=1}^{C+1}w^{k}_{c}\hat{y}^{k}_{r,c}\log
S_{r,c}^{k},$ (5)

where $w^{k}_{c}$ is the weight factor to smooth the learning process
following (Tang et al. 2017), $S^{k}\in\mathcal{R}^{R\times(C+1)}$ is computed
by $\frac{X^{\mathrm{fuse}}}{\lVert
X^{\mathrm{fuse}}\rVert}\frac{W^{r}}{\lVert W^{r}\rVert}$ and
$\hat{y}^{k}_{r,c}$ is the PGT labels of proposal $r$ for category $c$ in
branch $k$. And $\mathcal{L}_{\mathrm{reg}}^{k}$ is the smooth L1 loss (Ren et
al. 2015) in branch $k$.

## Experiments

Datasets. We evaluate the proposed WSOVOD framework on Pascal VOC 2007, 2012
(Everingham et al. 2010) and MS COCO (Lin et al. 2014), which are widely used
for WSOD. In addition, we also use ILSVRC (Russakovsky et al. 2015) and LVIS
(Gupta, Dollár, and Girshick 2019) for open-vocabulary learning, both of which
are widely used for FSOVOD.

Table 1:  Comparison with the state-of-the-art OVOD methods on MS COCO and
Pascal VOC.

Method Bakcbone Train Supervision COCO VOC07 Image-level Object-level
${AP_{N}}$ ${AP_{B}}$ ${AP}$ $m$AP ZS-LAB(Bansal et al. 2018) Incept.-Res. v2
– COCO 48 cls. 0.3 24.9 – – DELO(Zhu, Wang, and Saligrama 2020) DarkNet19 –
COCO 48 cls. 3.4 – 13.0 – PL(Rahman, Khan, and Barnes 2020) RN50-FPN – COCO 48
cls. 4.1 35.9 7.4 – SAN(Rahman, Khan, and Porikli 2020) RN50 – COCO 48 cls.
2.6 13.9 4.3 – BLC(Zheng et al. 2020) RN50 – COCO 48 cls. 4.5 42.1 8.2 – MS-
Zero(Gupta et al. 2020) RN101 – COCO 48 cls. – – $\mathbf{30.7}$ –
ContrastZSD(Yan et al. 2022) RN101 – COCO 48 cls. 6.3 45.1 11.1 –
SSB(Khandelwal et al. 2023) RN101 – COCO 48 cls. 10.2 $\mathbf{48.9}$ 16.9 –
RRFS(Huang et al. 2022) RN101 – COCO 48 cls. $\mathbf{13.4}$ 42.3 20.4 – OVR-
CNN(Zareian et al. 2021) RN50-C4 COCO caption COCO 48 cls. 22.8 46.0 39.9 52.9
ViLD(Gu et al. 2022) RN50-FPN CLIP400M COCO 48 cls. 27.6 59.5 51.3 – ZSD-
YOLO(Xie and Zheng 2022) CSP-DarkNet53 CLIP400M COCO 48 cls. 13.6 31.7 19.0 –
HierKD(Ma et al. 2022b) RN50-FPN Conceptual Captions COCO 48 cls. 20.3 51.3
43.2 – Detic(Zhou et al. 2022b) RN50-C4 COCO caption COCO 48 cls. 27.8 47.1
45.0 – Detic(Zhou et al. 2022b) RN50-C4 – COCO 48 cls. 1.3 – 39.3 –
RKDWTF(Bangalath et al. 2022) RN50-C4 COCO caption COCO 48 cls. 36.6 54.0 49.4
– SGDN(Shi, Hayat, and Cai 2023) RN50 Flickr30K, Visual Genome COCO 48 cls.
$\mathbf{37.5}$ $\mathbf{61.0}$ $\mathbf{54.9}$ – PBBL(Gao et al. 2022) RN50
COCO Caption, Visual Genome, SBU Caption COCO 48 cls. 30.8 46.1 42.1 59.2
WSOVOD RN50-WS-MRRP VOC07 – 15.4 7.8 9.8 63.4 WSOVOD RN50-WS-MRRP VOC12 – 17.0
9.3 11.3 64.8 WSOVOD RN50-WS-MRRP ILSVRC – 9.1 6.4 7.0 26.7 WSOVOD RN50-WS-
MRRP LVIS – 16.7 11.0 13.2 31.0 WSOVOD RN50-WS-MRRP COCO – $\mathbf{35.0}$
$\mathbf{27.9}$ $\mathbf{29.8}$ 60.9 WSOVOD RN50-WS-MRRP VOC07, VOC12 – 19.2
12.4 15.1 65.0 WSOVOD RN50-WS-MRRP COCO, VOC07, VOC12 – 35.4 27.3 29.8 65.0
WSOVOD RN50-WS-MRRP COCO, ILSVRC – 35.6 27.7 30.0 61.2 WSOVOD RN50-WS-MRRP
COCO, LVIS – $\mathbf{36.7}$ $\mathbf{28.4}$ $\mathbf{30.3}$ 62.3

Table 2:  LVIS experiments with RN18 on VOC07 and MS COCO. (“*” refers to
sampling by BCAS.)

Method Train Test PASCAL VOC 2007 MS COCO $m$AP CorLoc Avg. Precision, IoU:
0.5:0.95 0.5 0.75 WSOVOD LVIS 31.0 44.5 4.8 12.9 5.9 WSOVOD* LVIS 31.7 47.7
6.6 16.7 7.8 WSOVOD COCO 60.5 78.2 20.1 29.7 21.2 WSOVOD LVIS, COCO
$\mathbf{61.7}$ $\mathbf{79.3}$ $\mathbf{21.0}$ $\mathbf{30.1}$
$\mathbf{22.2}$

Table 3:  Comparison with the state-of-the-art WSOD methods on PASCAL VOC
2007, 2012 and MS COCO.

Method Supervision Bakcbone VOC 2007 VOC 2012 MS COCO $m$AP CorLoc $m$AP
CorLoc Avg. Precision, IoU: 0.5:0.95 0.5 0.75 WSDDN(Bilen and Vedaldi 2016)
$\mathcal{I}$ VGG16 34.8 53.5 – – 9.5 19.2 8.2 ContextLocNet(Kantorov et al.
2016) $\mathcal{I}$ VGG-F 36.3 55.1 35.3 54.8 11.1 22.1 10.7 OICR(Tang et al.
2017) $\mathcal{I}$ VGG16 41.2 60.6 37.9 62.1 – – – ML-LocNet(Zhang, Yang, and
Feng 2018) $\mathcal{I}$ VGG16 48.4 67.0 42.2 66.3 – 16.2 – MELM(Wan et al.
2018) $\mathcal{I}$ VGG16 47.3 61.4 42.4 – – – – C-MIL(Wan et al. 2019)
$\mathcal{I}$ VGG16 50.5 65.0 46.7 67.4 – – – PCL(Tang et al. 2018)
$\mathcal{I}$ VGG16 43.5 – – – 8.5 19.4 – WSOD2(Zeng et al. 2019)
$\mathcal{I}$ VGG16 53.6 69.5 47.2 71.9 10.8 22.7 – C-MIDN(Gao et al. 2019)
$\mathcal{I}$ VGG16 52.6 68.7 50.2 71.2 9.6 21.4 – MIST(Ren et al. 2020)
$\mathcal{I}$ VGG16 54.9 68.8 52.1 70.9 12.4 25.8 10.5 PG-PS(Cheng et al.
2020) $\mathcal{I}$ VGG16 51.1 69.2 48.3 68.7 – 20.7 – UWSOD(Shen et al.
2020a) $\mathcal{I}$ RN18-WS-MRRP 45.0 63.8 46.2 65.7 3.1 10.1 1.4 SPE(Liao et
al. 2022) $\mathcal{I}$ CaiT 51.0 70.4 – – 7.2 18.2 4.8 Seo et al.(Seo et al.
2022) $\mathcal{I}$ RN101 58.7 69.8 56.2 71.2 14.4 29.0 12.4 WSOVOD
$\mathcal{I}$ VGG16 59.1 77.2 59.8 79.7 18.8 27.1 19.7 WSOVOD $\mathcal{I}$
RN18-WS-MRRP 63.0 $\mathbf{80.6}$ 61.9 $\mathbf{81.0}$ 20.1 $\mathbf{29.7}$
21.2 WSOVOD $\mathcal{I}$ RN50-WS-MRRP $\mathbf{63.4}$ 80.1 $\mathbf{62.1}$
80.7 $\mathbf{20.5}$ 29.1 $\mathbf{21.4}$ Fast RCNN (Girshick 2015)
$\mathcal{O}$ VGG16 66.9 – 65.7 – 18.9 38.6 – Faster RCNN (Ren et al. 2015)
$\mathcal{O}$ VGG16 69.9 – 67.0 – 21.2 41.5 – WSOVOD $\mathcal{B}$ VGG16 67.2
88.2 65.4 84.5 16.4 31.1 15.3 WSOVOD $\mathcal{B}$ RN18-WS-MRRP 68.8 90.9 66.3
89.2 19.8 37.6 18.5 WSOVOD $\mathcal{B}$ RN50-WS-MRRP $\mathbf{71.8}$
$\mathbf{91.0}$ $\mathbf{69.7}$ $\mathbf{90.0}$ $\mathbf{21.6}$
$\mathbf{40.6}$ $\mathbf{20.8}$

Evaluation Metrics. Following the common setting of FSOVOD, we also split COCO
into $17$ novel classes and $48$ base classes, and use $AP_{N}$ and $AP_{B}$
to evaluate the results of $17$ novel classes and $48$ base classes,
respectively. We also use $AP$ to evaluate the results of $17+48$ total
classes. To compare the model performance in the WSOD setting, we use two
evaluation metrics: CorLoc and $m$AP. Correct localization (CorLoc) is a
commonly-used measurement that quantifies the localization performance by the
percentage of images that contain at least one object instance with at least
$50\%$ IoU to the ground-truth boxes. Mean average precision ($m$AP) follows
standard Pascal VOC protocol to report the $m$AP at $50\%$ IoU of the detected
boxes with the ground-truth ones. Furthermore, we report standard COCO metrics
for WSOD, including AP at different IoU thresholds.

Implementation Details. We use VGG16 (Simonyan and Zisserman 2015),
RN18/50-WS-MRRP (Shen et al. 2020b), initialized with the weights pre-trained
on ILSVRC as vision backbones. We use synchronized SGD training on Nvidia 3090
with a batch size of $4$, a mini-batch involving $1$ images per GPU. We use
learning rates of $1e^{-3}$ and $1e^{-2}$ for VGG16 and RN18/50-WS-MRRP
backbone, respectfully, a momentum of $0.9$, a dropout rate of $0.5$, a
learning rate decay weight of $0.1$. We fix the backbone weights for WSOD but
set a $1e^{-5}$ learning rate to backbones for OVOD.

### Open-Vocabulary and Cross-Dataset Detection

Since we are the first exploration for WSOVOD, we compare the proposed WSOVOD
framework with fully-supervised open-vocabulary object detection (FSOVOD).
Noted that FSOVOD divides the MS COCO categories into $48$ base and $17$ novel
classes (Bansal et al. 2018), and uses object-level annotations of $48$ base
classes during training. In addition, in order to expand vocabulary learning,
some works (Zareian et al. 2021; Zhou et al. 2022b; Zareian et al. 2021) use
weak annotation information including novel classes, such as tags, captions,
and etc. The first and second parts of Tab. 1 shows the state-of-the-art
FSOVOD results without and with image-level annotation, respectively. The
$6th$ row in the second part removes COCO captions in Detic (Zhou et al.
2022b), which results in a dramatic performance drop in novel classes with
only $1.3\%$ $AP_{N}$. This shows that fully-supervised methods are hard to
generalize well to detect novel classes if they lack the supervision
information of a large vocabulary. Therefore, it is necessary to study WSOVOD
on large-vocabulary datasets with only category annotations. As shown in the
third part of Tab. 1, WSOVOD exhibits strong generalization ability despite
large differences between train and test distributions. In particular, WSOVOD
significantly improves the $AP_{N}$ performance of novel classes compared to
FSOVOD with only object-level supervision. On COCO novel classes, WSOVOD even
surpasses FSOVOD methods, which require both image-level and object-level
supervision.

We further conduct experiments to train our WSOVOD with multiple datasets
jointly in the bottom part of Tab. 1. We observe that: (1) Cross-dataset
learning achieves superior or at least comparable results to the corresponding
single dataset. For instance, combing VOC07 and VOC12 significantly improves
the COCO APN with gains of $3.8\%$ and $2.2\%$ compared to separately using
VOC07 and VOC12 datasets, respectively. (2) Adding more image-level concepts
to COCO further improves the COCO APN. For instance, adding ILSVRC to COCO
performs better than adding VOC07 and VOC12 to COCO. (3) Adding denser image-
level annotations significantly improves results. For example, LVIS and COCO
share the same training set, and directly combining LVIS and COCO improves
$1.7\%$ APN, although the image-level labels in LVIS are incomplete.

### Rescuing Federated and Long-Tail Data

To better adjust and analyze our WSOVOD, we further conduct experiments on the
difficult federated and long-tail distribution LVIS (Gupta, Dollár, and
Girshick 2019) dataset, as shown in Tab. 2. When only using LVIS for training,
the performance of WSOVOD reaches saturation around $1$ epoch on VOC07 and
COCO. This is because LVIS is a federated dataset with sparse annotations. So,
image-level labels are not exhaustively annotated with all classes. The
missing classes are treated as background and generate incorrect supervision
signals. To this end, we introduce a batch-class-aware sampling, termed BCAS.
In BCAS, the data sampler first picks a category and then selects multiple
images containing that category to form a mini-batch. When equipped with BCAS
for LVIS, WSOVOD reaches saturation at about $3$ epochs and improves $3.8\%$
AP0.5 on COCO. We further add COCO to LVIS training without BCAS and observe
substantial performance improvements on VOC 2007 with gains of $30.7\%$ $mAP$
and $34.8\%$ CorLoc, respectively. Compared to single COCO, combing LVIS with
COCO also significantly improves the VOC 2007 $mAP$ from $60.5\%$ to $61.7\%$
and CorLoc from $78.2\%$ to $79.3\%$, respectively. This reveals that
incomplete image-level annotated data is helpful for WSOVOD cross-dataset
training.

### Weakly Supervised Object Detection

We compare our proposed method with the state-of-the-art WSOD methods on
VGG16, RN18-WS-MRRP, and RN50-WS-MRRP backbones. Tab. 3 shows the performance
comparisons on the Pascal VOC 2007, Pascal VOC 2012, and MS COCO, where
$\mathcal{I}$, $\mathcal{O}$ and $\mathcal{B}$ denote image-level supervision,
object-level supervision with class labels, and object-level supervision
without class labels, respectively. With the VGG16 backbone, the proposed
WSOVOD suppresses the performances of all previous WSOD methods for $mAP$ and
CorLoc on Pascal VOC and $AP_{0.5:0.95}$ on MS COCO. The proposed WSOVOD with
RN18-WS-MRRP backbone reaches the new state-of-the-art of $80.6\%$ and
$81.0\%$ CorLoc on Pascal VOC 2007 and 2012, respectively, and $29.7\%$ AP0.5
on MS COCO. With RN50-WS-MRRP backbone, WSOVOD further sets new state-of-the-
art of $63.4\%$ and $62.1\%$ $mAP$ on Pascal VOC 2007 and VOC 2012,
respectively, and $20.5\%$ $AP_{0.5:0.95}$ and $21.4\%$ AP0.75 on MS COCO.
Furthermore, with object-level supervision without class labels, our proposed
WSOVOD even shows comparable performance compared to FSOD in all datasets.

### Ablation Study

Table 4:  Ablation study of DAFE with RN18-WS-MRRP backbone on VOC 2007.

Train Data without DAFE with DAFE $m$AP CorLoc $m$AP CorLoc VOC07 62.6 78.7
63.0  ($\uparrow$ 0.4) 80.6  ($\uparrow$ 1.9) VOC07, VOC12 63.5 79.2 64.1
($\uparrow$ 0.6) 82.2  ($\uparrow$ 3.0) VOC07, COCO 61.4 78.2 63.0
($\uparrow$ 1.6) 80.5  ($\uparrow$ 2.3)

Table 5:  Ablation study of proposal generator with RN18-WS-MRRP backbone on
VOC 2007.

Proposal m$AP$ CorLoc LO-WSRPN 46.7  ($\uparrow$ 0.00) 65.1  ($\uparrow$ 0.00)
MCG 54.2  ($\uparrow$ 7.50) 71.9  ($\uparrow$ 6.80) SAM 61.7  ($\uparrow$
15.0) 77.5  ($\uparrow$ 12.4) LO-WSRPN + SAM 62.5  ($\uparrow$ 15.8) 79.9
($\uparrow$ 14.8) LO-WSRPN + SAM + refine 63.0  ($\uparrow$ 16.3) 81.0
($\uparrow$ 15.9)

Table 6:  Ablation study of proposal generator recall on VOC 2007 trainval on
IoU0.5 / average IoU0.5:1.0.

Method AR@1 AR@10 AR@100 AR@1,000 AR@2,000 LO-WSRPN 8.1/2.3 25.1/8.1 53.2/20.7
74.7/32.3 74.9/32.4 MCG $\mathbf{11.9}$/5.5 $\mathbf{39.1}$/18.0 70.0/37.6
89.5/54.8 93.0/58.8 SAM 8.7/$\mathbf{6.1}$ 35.3/$\mathbf{24.9}$
$\mathbf{80.9}$/$\mathbf{52.9}$ $\mathbf{92.1}$/$\mathbf{59.1}$ 92.1/59.1 LO-
WSRPN+SAM 8.1/2.3 25.1/8.1 53.8/21.0 86.2/46.5 97.1/63.2 LO-WSRPN+SAM+refine
8.5/4.6 30.7/11.2 65.5/32.7 88.4/52.5 $\mathbf{98.0}$/$\mathbf{65.2}$

We conducted three sets of ablation studies to investigate the effectiveness
of the proposed modules. We firstly ablate data-aware features extractor
(DAFE) in Tab. 4 to verify the effectiveness of DAFE for training multiple
datasets. We test all models on VOC07 test. When training on VOC12 and VOC07,
DAFE improves $m$AP by $0.6\%$ and CorLoc by $3.0\%$. Thus, DAFE significantly
improves the detection and localization performance, indicating that DAFE is
simple and effective. When training on COCO and VOC07, DAFE improves $m$AP by
$1.6\%$ and CorLoc by $2.3\%$. It demonstrates that DAFE also deals well with
the large distribution gap. DAFE also performs well on a single dataset. Thus,
introducing global image-level context to local proposal-level features is
helpful to WSOD. This reveals that DAFE not only successfully gathers dataset-
level bias but also image-level context. Secondly, we ablate the proposal
generator in Tab. 5. It shows that, as observed in (Shen et al. 2020a), only
using predictions from the model itself as supervision results in noisy
training, so that $m$AP and CorLoc only reach $46.7\%$ and $65.1\%$,
respectively. When using traditional object proposals, such as MCG (Pont-Tuset
et al. 2016), the performance is significantly improved, but compared with SAM
proposals based on high-level semantic information, it is still much worse.
When adding SAM proposals to LO-WSRPN proposals with the refinement mechanism,
our method improves $m$AP and CorLoc by $16.3\%$ and $15.9\%$, respectively.
Thirdly, we ablate the proposal generator on VOC07 trainval in terms of
proposal recall on IoU0.5 and average IoU0.5:1.0 in Tab. 6. We find that SAM
provides high-quality proposals, but it has too few proposals that are less
than $1,000$. The recall of proposals provided by traditional object
proposals, i.e., MCG, is relatively low on AR@$100$. And our LO-WSRPN achieves
the highest AR@$2,000$ with $98.0\%$ under IoU0.5 and $65.2$ under IoU0.5:1.0.

## Conclusion

In this paper, we propose a weakly supervised open-vocabulary object detection
framework, namely WSOVOD, which extends WSOD to detect and localize open-
vocabulary concepts and utilizes diverse and large-scale datasets with only
image-level annotation. On the widely used benchmark Pascal VOC and MS COCO,
the proposed WSOVOD achieves on-par or even better performance compared to
FSOVOD methods with box-level supervision. Moreover, WSOVOD sets new state-of-
the-art results for WSOD in both localization and detection tasks.

## Acknowledgments

This work was supported by National Key R$\&$D Program of China
(No.2022ZD0118202), the National Science Fund for Distinguished Young Scholars
(No.62025603), the National Natural Science Foundation of China (No.U21B2037,
No.U22B2051, No.62176222, No.62176223, No. 2176226, No.62072386, No.62072387,
No.62072389, No.62002305, NO. 62102151 and No.62272401), the Natural Science
Foundation of Fujian Province of China (No.2021J01002, No.2022J06001), and
CCF-Tencent Rhino-Bird Open Research Fund.

## References

  * Bahng et al. (2022) Bahng, H.; Jahanian, A.; Sankaranarayanan, S.; and Isola, P. 2022.  Exploring Visual Prompts for Adapting Large-Scale Models.  _arXiv_.
  * Bangalath et al. (2022) Bangalath, H.; Maaz, M.; Khattak, M. U.; Khan, S. H.; and Shahbaz Khan, F. 2022.  Bridging the gap between object and image-level representations for open-vocabulary detection.  In _NeurIPS_.
  * Bansal et al. (2018) Bansal, A.; Sikka, K.; Sharma, G.; Chellappa, R.; and Divakaran, A. 2018.  Zero-Shot Object Detection.  In _ECCV_.
  * Bilen and Vedaldi (2016) Bilen, H.; and Vedaldi, A. 2016.  Weakly Supervised Deep Detection Networks.  In _CVPR_.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-To-End Object Detection with Transformers.  In _ECCV_.
  * Cheng et al. (2020) Cheng, G.; Yang, J.; Gao, D.; Guo, L.; and Han, J. 2020.  High-Quality Proposals for Weakly Supervised Object Detection.  _TIP_.
  * Dietterich, Lathrop, and Lozano-Pérez (1997) Dietterich, T. G.; Lathrop, R. H.; and Lozano-Pérez, T. 1997.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles.  _AI_.
  * Dollár and Zitnick (2013) Dollár, P.; and Zitnick, C. L. 2013.  Structured Forests for Fast Edge Detection.  In _ICCV_.
  * Everingham et al. (2010) Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010.  The Pascal Visual Object Classes (voc) Challenge.  _IJCV_.
  * Felzenszwalb and Huttenlocher (2004) Felzenszwalb, P. F.; and Huttenlocher, D. P. 2004.  Efficient Graph-Based Image Segmentation.  _IJCV_.
  * Feng et al. (2022) Feng, C.; Zhong, Y.; Jie, Z.; Chu, X.; Ren, H.; Wei, X.; Xie, W.; and Ma, L. 2022.  PromptDet: Towards Open-Vocabulary Detection Using Uncurated Images.  In _ECCV_.
  * Gao et al. (2022) Gao, M.; Xing, C.; Niebles, J. C.; Li, J.; Xu, R.; Liu, W.; and Xiong, C. 2022.  Open Vocabulary Object Detection with Pseudo Bounding-Box Labels.  In _ECCV_.
  * Gao et al. (2019) Gao, Y.; Liu, B.; Guo, N.; Ye, X.; Wan, F.; You, H.; and Fan, D. 2019.  C-Midn: Coupled Multiple Instance Detection Network with Segmentation Guidance for Weakly Supervised Object Detection.  In _ICCV_.
  * Girshick (2015) Girshick, R. 2015.  Fast R-Cnn.  In _CVPR_.
  * Gu et al. (2022) Gu, X.; Lin, T.-Y.; Kuo, W.; and Cui, Y. 2022.  Open-Vocabulary Detection via Vision and Language Knowledge Distillation.  _ICLR_.
  * Gupta, Dollár, and Girshick (2019) Gupta, A.; Dollár, P.; and Girshick, R. 2019.  LVIS: A Dataset for Large Vocabulary Instance Segmentation.  In _CVPR_.
  * Gupta et al. (2020) Gupta, D.; Anantharaman, A.; Mamgain, N.; Balasubramanian, V. N.; Jawahar, C.; et al. 2020.  A multi-space approach to zero-shot object detection.  In _WCACV_.
  * Huang et al. (2022) Huang, P.; Han, J.; Cheng, D.; and Zhang, D. 2022.  Robust region feature synthesizer for zero-shot object detection.  In _CVPR_.
  * Jia et al. (2009) Jia, D.; Wei, D.; Socher, R.; Li, L.-J.; Li, K.; and Li, F.-F. 2009.  A Large-Scale Hierarchical Image Database.  In _CVPR_.
  * Jia et al. (2022) Jia, M.; Tang, L.; Chen, B.-C.; Cardie, C.; Belongie, S.; Hariharan, B.; and Lim, S.-N. 2022.  Visual Prompt Tuning.  In _ECCV_.
  * Jiang et al. (2022) Jiang, S.; Zhu, Y.; Liu, C.; Song, X.; Li, X.; and Min, W. 2022.  Dataset Bias in Few-shot Image Recognition.  _TPAMI_.
  * Kamath et al. (2021) Kamath, A.; Singh, M.; Lecun, Y.; Synnaeve, G.; Misra, I.; and Carion, N. 2021.  MDETR - Modulated Detection for End-To-End Multi-Modal Understanding.  In _ICCV_.
  * Kantorov et al. (2016) Kantorov, V.; Oquab, M.; Cho, M.; and Laptev, I. 2016.  Contextlocnet: Context-Aware Deep Network Models for Weakly Supervised Localization.  In _ECCV_.
  * Khandelwal et al. (2023) Khandelwal, S.; Nambirajan, A.; Siddiquie, B.; Eledath, J.; and Sigal, L. 2023.  Frustratingly Simple but Effective Zero-shot Detection and Segmentation: Analysis and a Strong Baseline.  _arXiv_.
  * Kim et al. (2020) Kim, D.; Lee, G.; Jeong, J.; and Kwak, N. 2020.  Tell Me What They’re Holding: Weakly-Supervised Object Detection with Transferable Knowledge from Human-Object Interaction.  In _AAAI_.
  * Kim, Lee, and Choo (2021) Kim, E.; Lee, J.; and Choo, J. 2021.  BiaSwap: Removing dataset bias with bias-tailored swapping augmentation.  In _ICCV_.
  * Kirillov et al. (2023) Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Dollár, P.; and Girshick, R. 2023.  Segment Anything.  In _ICCV_.
  * Li et al. (2022) Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; et al. 2022.  Grounded Language-Image Pre-Training.  In _CVPR_.
  * Liao et al. (2022) Liao, M.; Wan, F.; Yao, Y.; Han, Z.; Zou, J.; Wang, Y.; Feng, B.; Yuan, P.; and Ye, Q. 2022.  End-to-End Weakly Supervised Object Detection with Sparse Proposal Evolution.  In _ECCV_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft Coco: Common Objects in Context.  In _ECCV_.
  * Ma et al. (2022a) Ma, Z.; Luo, G.; Gao, J.; Li, L.; Chen, Y.; Wang, S.; Zhang, C.; and Hu, W. 2022a.  Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation.  In _CVPR_.
  * Ma et al. (2022b) Ma, Z.; Luo, G.; Gao, J.; Li, L.; Chen, Y.; Wang, S.; Zhang, C.; and Hu, W. 2022b.  Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation.  In _CVPR_.
  * Minderer et al. (2022) Minderer, M.; Gritsenko, A.; Stone, A.; Neumann, M.; Weissenborn, D.; Dosovitskiy, A.; Mahendran, A.; Arnab, A.; Dehghani, M.; Shen, Z.; et al. 2022.  Simple open-vocabulary object detection.  In _ECCV_.
  * Pont-Tuset et al. (2016) Pont-Tuset, J.; Arbelaez, P.; Barron, J. T.; Marques, F.; and Malik, J. 2016.  Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation.  _TPAMI_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning Transferable Visual Models from Natural Language Supervision.  In _ICML_.
  * Rahman, Khan, and Barnes (2020) Rahman, S.; Khan, S.; and Barnes, N. 2020.  Improved Visual-Semantic Alignment for Zero-Shot Object Detection.  In _AAAI_.
  * Rahman, Khan, and Porikli (2020) Rahman, S.; Khan, S. H.; and Porikli, F. 2020.  Zero-shot object detection: Joint recognition and localization of novel concepts.  _IJCV_.
  * Redmon et al. (2016) Redmon, J.; Divvala, S.; Girshick, R.; and Farhadi, A. 2016.  You Only Look Once: Unified, Real-Time Object Detection.  In _CVPR_.
  * Redmon and Farhadi (2017) Redmon, J.; and Farhadi, A. 2017.  YOLO9000: Better, Faster, Stronger.  In _CVPR_.
  * Ren et al. (2015) Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In _NeurIPS_.
  * Ren et al. (2020) Ren, Z.; Yu, Z.; Yang, X.; Liu, M.-Y.; Lee, Y. J.; Schwing, A. G.; and Kautz, J. 2020.  Instance-Aware, Context-Focused, and Memory-Efficient Weakly Supervised Object Detection.  In _CVPR_.
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015.  ImageNet Large Scale Visual Recognition Challenge.  _IJCV_.
  * Seo et al. (2022) Seo, J.; Bae, W.; Sutherland, D. J.; Noh, J.; and Kim, D. 2022.  Object discovery via contrastive learning for weakly supervised object detection.  In _ECCV_.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-Text Dataset for Automatic Image Captioning.  In _ACL_.
  * Shen et al. (2020a) Shen, Y.; Ji, R.; Chen, Z.; Wu, Y.; and Huang, F. 2020a.  UWSOD: Toward Fully-Supervised-Level Capacity Weakly Supervised Object Detection.  _NeurIPS_.
  * Shen et al. (2020b) Shen, Y.; Ji, R.; Wang, Y.; Chen, Z.; Zheng, F.; Huang, F.; and Wu, Y. 2020b.  Enabling Deep Residual Networks for Weakly Supervised Object Detection.  In _ECCV_.
  * Shi, Hayat, and Cai (2023) Shi, H.; Hayat, M.; and Cai, J. 2023.  Open-Vocabulary Object Detection via Scene Graph Discovery.  In _ACMMM_.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_.
  * Sun et al. (2021) Sun, Z.; Cao, S.; Yang, Y.; and Kitani, K. M. 2021.  Rethinking Transformer-Based Set Prediction for Object Detection.  In _ICCV_.
  * Tang et al. (2018) Tang, P.; Wang, X.; Bai, S.; Shen, W.; Bai, X.; Liu, W.; and Yuille, A. 2018.  Pcl: Proposal Cluster Learning for Weakly Supervised Object Detection.  _TPAMI_.
  * Tang et al. (2017) Tang, P.; Wang, X.; Bai, X.; and Liu, W. 2017.  Multiple Instance Detection Network with Online Instance Classifier Refinement.  In _CVPR_.
  * Tian et al. (2019) Tian, Z.; Shen, C.; Chen, H.; and He, T. 2019.  Fcos: Fully convolutional one-stage object detection.  In _ICCV_.
  * Torralba and Efros (2011) Torralba, A.; and Efros, A. A. 2011.  Unbiased Look at Dataset Bias.  In _CVPR_.
  * Uijlings et al. (2013) Uijlings, J. R.; Van De Sande, K. E.; Gevers, T.; and Smeulders, A. W. 2013.  Selective Search for Object Recognition.  _IJCV_.
  * Wan et al. (2019) Wan, F.; Liu, C.; Ke, W.; Ji, X.; Jiao, J.; and Ye, Q. 2019.  C-Mil: Continuation Multiple Instance Learning for Weakly Supervised Object Detection.  In _CVPR_.
  * Wan et al. (2018) Wan, F.; Wei, P.; Jiao, J.; Han, Z.; and Ye, Q. 2018.  Min-Entropy Latent Model for Weakly Supervised Object Detection.  In _CVPR_.
  * Xie and Zheng (2022) Xie, J.; and Zheng, S. 2022.  Zero-shot Object Detection Through Vision-Language Embedding Alignment.  In _ICDMW_.
  * Yan et al. (2022) Yan, C.; Chang, X.; Luo, M.; Liu, H.; Zhang, X.; and Zheng, Q. 2022.  Semantics-guided contrastive network for zero-shot object detection.  _TPAMI_.
  * Yang, Li, and Dou (2019) Yang, K.; Li, D.; and Dou, Y. 2019.  Towards Precise End-To-End Weakly Supervised Object Detection Network.  In _ICCV_.
  * Zang et al. (2022) Zang, Y.; Li, W.; Zhou, K.; Huang, C.; and Loy, C. C. 2022.  Open-Vocabulary DETR with Conditional Matching.  In _ECCV_.
  * Zareian et al. (2021) Zareian, A.; Rosa, K. D.; Hu, D. H.; and Chang, S.-F. 2021.  Open-Vocabulary Object Detection Using Captions.  In _CVPR_.
  * Zeng et al. (2019) Zeng, Z.; Liu, B.; Fu, J.; Chao, H.; and Zhang, L. 2019.  Wsod2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-Supervised Object Detection.  In _ICCV_.
  * Zhang, Yang, and Feng (2018) Zhang, X.; Yang, Y.; and Feng, J. 2018.  Ml-Locnet: Improving Object Localization with Multi-View Learning Network.  In _ECCV_.
  * Zheng et al. (2021) Zheng, M.; Gao, P.; Zhang, R.; Li, K.; Wang, X.; Li, H.; and Dong, H. 2021.  End-To-End Object Detection with Adaptive Clustering Transformer.  In _BMVC_.
  * Zheng et al. (2020) Zheng, Y.; Huang, R.; Han, C.; Huang, X.; and Cui, L. 2020.  Background learnable cascade for zero-shot object detection.  In _ACCV_.
  * Zhou et al. (2022a) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022a.  Conditional Prompt Learning for Vision-Language Models.  In _CVPR_.
  * Zhou et al. (2022b) Zhou, X.; Girdhar, R.; Joulin, A.; Krähenbühl, P.; and Misra, I. 2022b.  Detecting Twenty-thousand Classes using Image-level Supervision.  In _ECCV_.
  * Zhu, Wang, and Saligrama (2020) Zhu, P.; Wang, H.; and Saligrama, V. 2020.  Don’t even look once: Synthesizing features for zero-shot detection.  In _CVPR_.
  * Zhu et al. (2021) Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2021.  Deformable Detr: Deformable Transformers for End-To-End Object Detection.  In _ICLR_.