11institutetext: East China Normal University, Shanghai, China
22institutetext: Holoflow Digital Technology, Hangzhou, China
33institutetext: Harbin Institute of Technology, Shenzhen, China

# Overview of the PromptCBLUE Shared Task in CHIP2023

Wei Zhu 11 Xiaoling Wang Corresponding author. Email: . 11
<EMAIL_ADDRESS>Mosha Chen 22 Buzhou Tang
33

###### Abstract

This paper presents an overview of the PromptCBLUE shared task111http://cips-
chip.org.cn/2023/eval1 held in the CHIP-2023 Conference. This shared task
reformualtes the CBLUE benchmark, and provide a good testbed for Chinese open-
domain or medical-domain large language models (LLMs) in general medical
natural language processing. Two different tracks are held: (a) prompt tuning
track, investigating the multitask prompt tuning of LLMs, (b) probing the in-
context learning capabilities of open-sourced LLMs. Many teams from both the
industry and academia participated in the shared tasks, and the top teams
achieved amazing test results. This paper describes the tasks, the datasets,
evaluation metrics, and the top systems for both tasks. Finally, the paper
summarizes the techniques and results of the evaluation of the various
approaches explored by the participating teams.

###### Keywords:

PromptCBLUE Large language models Medical natural language processing
parameter efficient fine-tuning in-context learning.

##  1 Introduction

In 2023, the launch of large language models like ChatGPT, gpt-4, Claude, Bard
have taken the world by surprise. People are amazed by their general
capabilities: (a) universal NLP task solver [38]. (b) powerful chatting
capabilities. (c) following human instructions [35, 3]. (d) reasoning and
planning capabilities [48]. Recent works have shown that these powerful LLMs
have expert level knowledge in the medical domain [43]. Although these models
are powerful, they are proprietary, and can not be deployed locally,
preventing many applications that have serious privacy concerns [52].
Recently, more and more powerful open-sourced LLMs are being released [30],
making it more and more convenient for developers to train or fine-tune a
local LLM for application developments.

In order to evaluate the multitask capabilities of the recent open-sourced
Chinese LLMs in medical NLP tasks, we re-build the CBLUE benchmark [56] into
PromptCBLUE [75], a large-scale multi-task prompt tuning benchmark dataset
(see Figure 1). This benchmark is one of first and largest prompt tuning
benchmark for medical LLMs in Chinese. The PromptCBLUE shared task is held in
the 2023 China Health Information Processing Conference, with the help of the
conference committees. The shared task evaluate Chinese LLMs in two aspects,
corresponding to two tracks: (a) the Parameter-efficient Fine-tuning (PEFT)
Track, and (b) the In-Context Learning (ICL) Track.

Figure 1: We introduce PromptCBLUE, a large-scale instruction tuning benchmark
for Chinese medical LLMs, which converts different types of medical natural
language processing tasks into a unified prompt-response generation task.
PromptCBLUE consists of five cohorts of 18 tasks, which cover a variety of
medical applications.

The PEFT track is closely related to the recent trend of PEFT in the LLM
research. Despite LLMs becoming general task solvers, fine-tuning still plays
a vital role in efficient LLM inference and controlling the style of the LLMs’
generated contents.222Recently, OpenAI also released the fine-tuning API for
GPT-3.5-turbo. See blog post: https://openai.com/blog/gpt-3-5-turbo-fine-
tuning-and-api-updates. Fine-tuning such large models by full parameters is
prohibitive since it requires a large amount of GPU memory and computations.
Thus, parameter-efficient fine-tuning (PEFT) [59, 61] has raised much
attention in the research field since in PEFT, the tunable parameters are
often less than 1% of the LLMs and the computation costs will be significantly
decreased. In the PEFT Track, participants are asked to fine-tune a given
open-sourced Chinese LLM with a single PEFT module for all the 18 sub-tasks of
PromptCBLUE, while keeping the backbone LLM unchanged. This track intends to
challenge the participants to come up with novel PEFT modules or novel multi-
task training methods.

With the recent advancements in scaling up model parameters, large language
models (LLMs) showcase promising results on a variety of few-shot tasks
through in-context learning (ICL), where the model is expected to directly
generate the output of the test sample without updating parameters. This is
achieved by conditioning on a manually designed prompt consisting of an
optional task description and a few demonstration examples [39]. Since then,
many efforts have been made on ICL [22]. In the ICL Track of , participants
are asked to push the limit of ICL for medium sized (6B, 7B or 13B parameters)
open-sourced LLMs. The backbone LLMs are freezed and they are not allowed to
introduce any additional parameters to the backbone models. However, they can
improve the LLMs’ performance by designing better prompts, especially select
proper demonstrations for any given samples. A BERT base model or any model
with a similar size is allowed to facilitate the demonstration selection.

The PromptCBLUE shared task has raised much attention in both the industry and
academia. A total of 600 teams have participated in either track of the shared
task, and 48 teams have submitted predictions for the second round. In this
paper, we will review the shared task, the winning teams and their
methodologies, and discuss future research directions.

##  2 Related Work

###  2.1 Medical natural language processing

The developments in neural networks and natural language processing has
advanced the field of medical natural language processing (MedNLP) [63, 11,
70, 64, 68, 72, 62]. In the pre-BERT era, firstly, RNNs like LSTM/GRU are used
for processing sequential medical data such as text and speech [1].
Convolutional networks are also used for medical text classificaiton [14, 65].
The techniques of Graph neural networks are also explored for diagnose
recommendations [24]. In this period, many different model architectures are
specially designed for better performances on a specific MedNLP task [70, 74,
60]. Since BERT [6], the pretrained language models (PLMs) become the deafult
solution for MedNLP. In this stage, researcher becomes less interested in
modifying the model architecture, but instead trying to pretrain or further
pretrain a PLM from the open domain to the medical domain [10, 66, 8, 67, 47].

With the wide study of LLMs, the field of MedNLP is also being revolutionized.
There are already works on adapting LLM backbones to the medical domain
question answering [73]. And [75] propose PromptCBLUE, a prompt learning based
benchmark dataset for examing the LLMs’ ability in MedNLP tasks. This work
investigates the capabilities of both the commercial LLMs like ChatGPT, and
the open-sourced ones by employing the PromptCBLUE as the testbed, providing a
deeper understanding of LLMs for future MedNLP research.

###  2.2 Parameter-efficient Fine-tuning

In this subsection, we review currently the most popular PEFT methods.

Adapter-based tuning. One of the most important research lines of PEFT is
adapter-based tuning. Adapter [13] inserts adapter modules with bottleneck
architecture between every consecutive Transformer [45] sublayers.
AdapterFusion [36] only inserts sequential adapters after the feed-forward
module. Adapter-based tuning methods have comparable results with model tuning
when only tuning a fraction of the backbone model’s parameter number. Due to
their strong performance, a branch of literature has investigated the
architecture of adapters in search of further improvements. [12] analyze a
wide range of PETuning methods and show that they are essentially equivalent.
They also propose the general architecture of PEFT, and derive the Parallel
Adapter which connects the adapter modules in parallel to the self-attention
and MLP modules in the Transformer block. AdapterDrop [42] investigates the
efficiency of removing adapters from lower layers. Adaptive adapters [34]
investigate the activation functions of adapters and propose to learn the
activation functions of adapters via optimizing the parameters of rational
functions as a part of the model parameters. Compacter [32] uses low-rank
parameterized hypercomplex multiplication [18] to compress adapters’ tunable
parameters. LST [44] improves the memory efficiency by forming the adapters as
a ladder along stacked Transformer blocks, and it enhances the adapter module
by adding a self-attention module to its bottleneck architecture. [44, 15, 58]
try to add different encoding operations, like self-attention operations and
convolutions between the bottleneck structure of adapters, and achieve better
performances. Learned-Adapter [59] builds upon the above adapter-based methods
and enhance the performance of adapter tuning by automatically learning better
architectures for adapters.

Prompt tuning methods Prompt tuning [19] and P-tuning [29] insert a soft
prompt to word embeddings only, and can achieve competitive results when
applied to supersized PTMs. Prefix-tuning [21] and P-tuning v2 [28] insert
prompts to every hidden layer of PTM. IDPG [49] uses the prompt generator with
parameterized hypercomplex multiplication [18] to generate a soft prompt for
every instance. LPT [27] improves upon IDPG by selecting an intermediate layer
to start inserting prompts. SPT [71] designs a mechanism to automatically
decide which layers to insert new instance-aware soft prompts.

Literature for the LoRA methods Since LoRA is the most popular PEFT method in
the era of large language models, there are many works that are orthogonal to
AdaLoRA, SoRA and our work that are devoted to improve LoRA on many different
aspects. QLoRA [5] proposes a novel quantization method that can significantly
reduce the memory consumptions of LLMs during LoRA fine-tuning. LoRA-FA [55]
freezes parts of the randomly initialized LoRA matrices. (d) VERA [17]
investigate whether one could froze the randomly initialized LoRA matrices and
only learns a set of scaling vectors. Tying LoRA matrices across layers are
also investigated by VERA.

###  2.3 In-context learning

GPT-3 [39], the OpenAI’s former SOTA LLM, raise the attention of the research
field to a novel research direction: in-context learning (ICL). In their
paper, they have showcased that GPT-3, a self-supervised pretrained model, can
immediately master a new task it is never trained on by reading a manually
designed prompt consisting of an optional task description and a few
demonstration examples. Since then, a branch of literature has been devoted to
investigate different aspects of ICL. A series of theoretical analysis
attempted to understand why ICL works [16, 33, 4]. [4] explains language
models as meta-optimizers and understand in-context learning as implicit fine-
tuning. They prove that GPT first produces meta-gradients according to the
demonstration examples, and then these meta-gradients are applied to the
original GPT to build an ICL model. Since performance of ICL has been shown to
be highly sensitive to the selection of demonstration examples [22]. [41]
proposed to learn to retrieve demonstration examples. [22] proposes a series
of techniques to enhance the performance of demonstration selection. [20]
selected diverse demonstrations to improve in-context compositional
generalization. More recent studies have explored leveraging the output
distributions of language models to select few-shot demonstrations [23]. [37]
iteratively selects examples that are diverse but still strongly correlated
with the test sample as ICL demonstrations.

##  3 Overview of PromptCBLUE

###  3.1 overview

Built upon the CBLUE benchmark [56], we created an extensive multi-task test
suite in the medical domain for the LLMs that supports Chinese. The tasks in
PromptCBLUE can be divided into the following groups:

  * •

Medical information extraction, including: (a) CMeEE-V2 [53], a medical named
entity recognition task. (b) CMeIE [9], a medical triple extraction task. (c)
CHIP-CDEE, which asks models to extract clinical finding events; (d) CHIP-CDN,
which asks one to map the diagnosis descriptions to standard ICD-10333
https://www.whofic.nl/familie-van-internationale-classificaties/referentie-
classificaties/icd-10 disease terms. (e) IMCS-V2-NER for extracting medical
entities in the dialogues from the medical dialogue datasets IMCS-V2 [2]. (f)
Text2DT, which is a complex medical information extraction task proposed by
[69]. This novel task asks a model to extract medical decision processes in
the format of binary trees from the unstructured medical text. (g) CmedCausal
[25], which asks a model to extract triplets with causal relations from
unstructured medical documents.

  * •

Medical text classification, which includes: (a) CHIP-CTC [76], the
classification task for Chinese eligibility criteria in clinical trials. (b)
KUAKE-QIC, which classifies online medical queries. (c) IMCS-V2-DAC for
medical intent classification of medical dialogues.

  * •

Medical natural language inference tasks, including CHIP-STS [31], KUAKE-QQR,
KUAKE-IR, and KUAKE-QTR tasks, which asks a model to determine the semantic
relations between a pair of medical queries or a query-document pair.

  * •

Symptom status understanding for medical dialogues. Medical dialogues, like
medical consultations, are centered around the patients’ symptoms. However,
not every symptom mentioned in the dialogues is related to patients or
reflects patients’ current medical status. Thus, to gain a deep understanding
of the doctor-patient dialogues, a model must be able to extract the symptoms
and their statuses. This cohort includes (a) IMCS-V2-SR and (b) CHIP-MDCFNPC
for extracting clinical findings and their status. This cohort of tasks is
based on the medical dialogue datasets IMCS-V2 [2] and CHIP-MDCFNPC [51].

  * •

Medical content generation, which includes two generation tasks based on
medical dialogues between patients and doctors: (a) IMCS-V2-MRG [2], which
asks LLMs to summarize the medical dialogues. (b) MedDG [26], which asks LLMs
to act like a doctor and respond to the patient’s queries in a dialogue.

###  3.2 Prompt collection

We employ both manual efforts and ChatGPT to collect prompt templates for
PromptCBLUE. The prompt templates mainly contain task instructions describing
what we want the LLMs to provide, given the text input. Firstly, each of the
three annotators who are graduate students majoring in computer science and
are studying LLMs will write around two seed prompt templates manually. We ask
the annotators to write the prompts as diversified as possible. The prompt
templates are then reviewed by a panel of two medical experts and a senior NLP
researcher to ensure their validity. If a prompt is not proper (for example,
not expressing the task clearly), we will ask the annotators to modify it
until the domain experts accept it. Then, we will ask the ChatGPT to rephrase
each of the six seed prompts templates ten times without changing the meaning
or changing the placeholders in the templates. Then, the generated templates
will be reviewed by the same panel of experts, and only the templates passing
the reviews will be added to the template pool. After the prompt collection
process, there are a total of 320 prompt templates in PromptCBLUE.444The
manually written prompt templates and the augmented template sets are open-
sourced at our code repository, https://github.com/michael-wzhu/PromptCBLUE.

###  3.3 Response format

Note that LLMs can only generate token sequences to represent their answers to
the queries. We have to transform the structured outputs of the original CBLUE
tasks into natural language sequences. In the Appendix, we present the target
output formats for each task under PromptCBLUE.

Recently, [48, 46] show that LLMs’ reasoning or task-solving ability can be
improved if LLMs are asked to complete the task step-by-step or reason step-
by-step, that is, chain-of-thought (COT) prompting. In this task, inspired by
[48], we will explore the idea of COT in the design of prompt and response
formats for complex medical information extraction tasks. Take CMeIE as an
example. With COT, in the prompt, we will ask the LLM to determine the
existing relations in the given sentence and then extract the triples for each
relation. Accordingly, in the response, first, the relations will be
presented, and then the extracted triples will be organized into relation
groups. Without COT, the prompt will not instruct the LLM to solve the task
step-by-step, and in the response, the triples will be generated one by one.
In this work, unless stated otherwise, we will use the prompt and response
formats with COT for the CMeEE-V2, CMeIE, CHIP-CDEE, and IMCS-V2-NER tasks. We
will use ablation studies to demonstrate that COT benefits in-context learning
of LLMs like ChatGPT and fine-tuned open-sourced LLMs.

###  3.4 Sample format

In PromptCBLUE, all the samples from different tasks are organized in the
following data structure:

    {
      "input": str,
      "target": str,
      "answer_choices": Union[list, Nonetype],
      "sample_id": str,
      "task_type": str,
      "task_dataset": str,
    }

Here, _input_ is the prompt sequence, _target_ is the response we want the
model to output, or at least the model should generate responses that contain
the answers to the prompt. The other four keys are auxiliary, and the LLMs
will not use them. _sample_id_ is the sample index. _answer_choices_ is the
label option allowed by the prompt. The value for this key will be _None_ if
the task does not have a predefined label set. _task_dataset_ specifies the
task name in the original CBLUE benchmark, and _task_type_ is the original
CBLUE task type.

###  3.5 Dataset splits

Note that after filling in the CBLUE data samples in the prompt templates, the
train/dev/test sets for some tasks will be quite large. Considering that LLMs
require high computation resources to fine-tune and have large latency for
token sequence generation, we limit the training samples of each task to 3000
to 5000 and the dev/test set to 400. We first fill each prompt template with
the samples to construct a large test sample pool and randomly select the
prompt-response pairs via uniform sampling.555Note that the training set is
provided just as a reference for participants in the Tianchi competition since
some of the original CBLUE tasks have large training sets. In Table 1, we
present the dataset statistics for the PromptCBLUE dataset.

Table 1: Dataset statistics for the PromptCBLUE benchmark.  Task | #Train/#dev/#test | Prompt length | target length
---|---|---|---
_Sub-tasks_
CMeEE-V2 | 5000/400/400 | 107.88 | 54.03
CMeIE | 5000/400/400 | 293.72 | 135.51
CHIP-CDEE | 3000/400/400 | 142.61 | 180.93
CHIP-CDN | 5000/400/400 | 281.79 | 10.37
CHIP-CTC | 6600/704/704 | 214.61 | 3.81
CHIP-STS | 5000/400/400 | 66.26 | 2.0
KUAKE-QIC | 5500/440/440 | 81.58 | 4.09
KUAKE-QTR | 5000/400/400 | 96.38 | 7.23
KUAKE-QQR | 5000/400/400 | 89.38 | 7.61
KUAKE-IR | 5000/400/400 | 203.33 | 2.78
CHIP-MDCFNPC | 5000/400/400 | 744.99 | 67.67
IMCS-V2-SR | 5000/400/400 | 137.13 | 36.33
IMCS-V2-NER | 5000/400/400 | 61.66 | 23.65
IMCS-V2-DAC | 5000/512/512 | 371.62 | 8.56
IMCS-V2-MRG | 3000/400/400 | 821.1 | 105.08
MedDG | 5000/400/400 | 194.75 | 27.71
Text2DT | 1500/400/400 | 465.10 | 392.47
CMedCausal | 3000/400/400 | 423.68 | 272.25
_Total_
PromptCBLUE | 87100/8456/8456 | 265.22 | 71.10

Quality Checking The quality check of our data is conducted in the following
three aspects:

  * •

Ensuring the quality of the prompt templates with the help of the expert
panel, as described above.

  * •

Checking the quality of the CBLUE benchmark. During the development of the
PromptCBLUE benchmark, we are also helping the CBLUE benchmark to improve the
annotations. For example, we have found that the original QQR task has an
inappropriate label set and asked the CBLUE organizers to re-annotate this
task.

  * •

Random sampling. To ensure the data quality, we sampled 5% or 200 of the
samples from each task of PromptCBLUE, and each sample was examined by a group
of annotators from the medical field. Finally, we identify an average of 0.9%
mislabeling rate. Based on the evaluation results in the next section, such an
error rate will not significantly impact the overall evaluation accuracy.

##  4 Participating Teams and Methods

###  4.1 Participating Teams

This shared task is held in CHIP-2023 Conference as Shared Task 1. We held
this shared task with the help the Tianchi Team, and all the logistics are
handled by the Tianchi platform.666The PEFT track is at
https://tianchi.aliyun.com/competition/entrance/532132, and the ICL track is
held at https://tianchi.aliyun.com/competition/entrance/532131.  In the first
round of the shared task, 362 teams participated in the PEFT track, and 238
teams participated in the ICL track. In the second round, 31 teams submitted
predictions for the PEFT track, and 17 teams submitted predictions for the ICL
track.

###  4.2 Wining teams

For both tracks, we rank the teams by the average score of all the 18 sub-
tasks in the PromptCBLUE benchmark. In accordance with the regulations of the
shared tasks in CHIP-2023, the top-3 teams of each track are eligible for the
awards and certificates issued by the CHIP-2023 committee. And these teams are
invited to submit papers to share their techniques and experiences to the
community.777The third place of the PEFT track is kept empty since the
corresponding team does not submit their materials for review.

In the table 2 below, we have listed the top-3 teams from each track. We can
see that there is a clear gap between the two tracks with respect to the
average score, showing that the open-sourced LLMs can not perform as well as
fine-tuning under the ICL setting.

Table 2: The winning teams and their test results of the PromptCBLUE shared task.  Team name |  Rank |  Institution |  Avg Score
---|---|---|---
Winners of the PEFT track
pt.boys |  1 |  Huimei Healthcare Management Services (惠每科技) |  71.38
练习一下 |  2 |  苏州大学 (University of Suzhou) & 北京邮电大学 (Beijing University of Posts and Telecommunications) |  68.09
Winners of the ICL track
紫丁香队 |  1 |  哈尔滨工业大学 (Harbin Institute of Technology) |  40.27
ECNU-LLM |  2 |  华东师范大学 (East China Normal University) |  39.24
IMI1214 |  3 |  上海理工大学 (University of Shanghai for Science and Technology) & 南京大学 (Nanjing University) |  36.80

###  4.3 Methods of the PEFT track

In this subsection, we will analyze the methods adopted by the top ranked
teams in the PEFT track.

Pre-trained backbones In this shared task, to ensure fair comparisons and not
to turn the whole task to a search of stronger LLM backbone, we only allow the
following open-sourced Chinese LLM backbone models: (a)
ChatGLM-6B-2888https://huggingface.co/THUDM/chatglm2-6b.; (2)
Baichuan-13B-Chat999https://huggingface.co/baichuan-inc/Baichuan-13B-Chat.;
(3) Chinese-LlaMA2-7B-chat101010https://huggingface.co/michaelwzhu/Chinese-
LlaMA2-chat-7B-sft-v0.3.; (4) Chinese-
LlaMA2-13B-chat111111https://huggingface.co/michaelwzhu/Chinese-
LlaMA2-13B-chat.. From the results of the winning teams in the PEFT track, we
can see that: (a) the 13B models perform better 7B models under PEFT tuning;
(b) the Baichuan-13B-Chat model obtains the strongest performance.

Data processing and augmentation Note that our PromptCBLUE benchmark comes
with a training set, and naturally it would enhance the model performance if
we conduct certain data augmentation operations on the training set. The
following are the data augmentation operations investigated by the winning
teams:

  * •

Note that our PromptCBLUE benchmark comes with a training set containing at
most 5k to 6.6k samples for each sub-task. However, some tasks from the
original CBLUE benchmark have a large training set. Thus, we can augment the
training set with the original CBLUE ones. For example, Team pt.boys augments
the training sets of the CHIP-CDEE, CMeEE-V2, CHIP-CDN, CMeIE-V2, IMCS-V2-NER,
CHIP-MDCFNPC, CHIP-CTC, CHIP-STS and CMedCausal sub-tasks with the CBLUE data
by employing the prompt templates open-sourced by us121212The prompt templates
of the PromptCBLUE benchmark are open-sourced at https://github.com/michael-
wzhu/PromptCBLUE/blob/main/src/data/templates_augment.json. . They augment
each of the above mentioned sub-task’s training sample size to 16000. To
ensure task samples’ balance, they also up-sample the other sub-tasks’ sample
sizes to 8000.

  * •

The previous literature [70, 7, 54, 57] shows that augmenting training samples
by randomly masking the contents of the input sentence can improve the
robustness of the trained models. Team pt.boys randomly replace the token in
the sentence with the <unk> token, for the sentence classification tasks in
PromptCBLUE (IMCS-V2-DAC, CHIP-CTC, CHIP-STS, KUAKE-IR, KUAKE-QIC, KUAKE-QQR,
and KUAKE-QTR)

The parameter-efficient fine-tuning methods The PEFT methods rises ffollowing
the rise of pre-trained language models, especially when the research field
made efforts to models with larger scales. There are a variety of PEFT methods
in the pre-LLM era. However, since the rise of LLMs, especially the open-
sourced LLMs, LoRA becomes the most popular PEFT method. We believe that the
popularity of LoRA comes from the following three reasons: (a) LoRA is a
reparameterization of the original Transformer weights, thus it can be merged
to the model backbone and introduce no additional latency; (b) LoRA has a good
theoretical basis: it comes from the idea of intrinsic space, and the fact
that fine-tuning a well pretrained model is in essence low-rank. (c) open-
source implementations. LoRA is implemented by many open-sourced code
repositories like the Huggingface PEFT
package131313https://github.com/huggingface/peft., thus it is very convenient
to use. In the PEFT track, all the winning teams use the LoRA method to fine-
tune the LLM backbones.

Training techniques In order to fine-tune the model properly, a few issues
should be addressed. First, despite the fact that most the LLM backbone is
freezed and we only update the LoRA parameters, the LLMs still requires a
large GPU memory, which may be difficult to obtain for academia. Thus, Team
"联系一下" employ the QLoRA method [5] to reduce the memory requirements during
fine-tuning. QLoRA proposes a novel 4-bit quantization method that proves to
have minimum effects for the model fine-tuning. In addition, Flash Attention
is also applied to reduce the memory comsumption when dealing with long
sequences. Second, since we are dealing with a multi-task prompt tuning task,
many teams pay attention to balance the sample sizes of the subtasks, so that
the fine-tuned LLM will perform propoerly on each subtask.

###  4.4 Methods of the ICL track

In this subsection, we will analyze the winning teams’ techniques in the ICL
track.

Pre-trained backbones From the results of the winning teams in the PEFT track,
we can see that: (a) similar to the PEFT track, the 13B models perform better
7B models in ICL; (b) Chinese-LlaMA2-13B-chat helps to achieves the best ICL
results, showing that Baichuan-13B-Chat is not dominating in every aspects.

Demonstration selection The demonstration selection method is the core of the
ICL track. When the LLM is a blackbox and can not be fine-tuned, such as model
APIs, ICL capabilities play an important role for unseen or emergent tasks.
All three winning teams have used the similarity based demonstration
selection. That is, for a test prompt, a sentence embedding model is employed
to retrieve the most similar prompts in the training set. When computing a
similarity score between the test prompt and training sample, different
approaches can be employed: (a) traditional methods like BM25 [40]. (b)
semantic representation, which relies on a model to transformer the sequences
to hidden vectors. Team "紫丁香队" used both methods, and the other two teams
mainly rely on the semantic models. Three teams apply different sentence
embdding models. For example, Team ECNU-LLM applied the BGE base model [50]
for semantic representations and semantic retrieval. When the similar samples
are retrieved, there are many approaches to determine the final demonstration
combinations:

  * •

The greedy approach. The retrieved top 3-10 training samples are used as
demonstrations.

  * •

Knapsack based demonstration selection. Since we are using the LLMs for
inference, we have to consider the maximum length it can handle given our GPU
environment. Thus, when we have a pre-defined maximum length, choosing the
combination of demonstrations becomes a classic knapsack problem if we
consider the similarity score of a demonstration as the value, and the
sample’s length as the item weight. Team "紫丁香队" has investigated this approach
and find that this strategy is better than the greedy method on four of the
subtasks.

##  5 Conclusion

In this article, we review the PromptCBLUE benchmark, the first large-scale
Chinese medical prompt tuning benchmark. Then, we given an overview of the
PromptCBLUE shared task in the CHIP-2023 conference. The shared task is a huge
success, attracting participants from both the industry and academia. Then, we
analyze the winning methods of the two tracks. Different techniques are
investigated in the shared task to fully uncover the limit of the Chinese
LLMs.

##  6 Acknowledgements

This work was supported by NSFC grants (No. 61972155 and 62136002) and
National Key R&D Program of China (No. 2021YFC3340700), and Shanghai Trusted
Industry Internet Software Collaborative Innovation Center.

## References

  * [1] Beeksma, M., Verberne, S., van den Bosch, A., Das, E., Hendrickx, I., Groenewoud, S.: Predicting life expectancy with a long short-term memory recurrent neural network using electronic medical records. BMC medical informatics and decision making 19(1), 1–15 (2019)
  * [2] Chen, W., Li, Z., Fang, H., Yao, Q.Q., Zhong, C., Hao, J., Zhang, Q., Huang, X., Peng, J., Wei, Z.: A benchmark for automatic medical consultation system: frameworks, tasks and datasets. Bioinformatics 39 (2022), https://api.semanticscholar.org/CorpusID:248239674
  * [3] Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., Sun, M.: Ultrafeedback: Boosting language models with high-quality feedback. ArXiv abs/2310.01377 (2023), https://api.semanticscholar.org/CorpusID:263605623
  * [4] Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., Wei, F.: Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Findings of the Association for Computational Linguistics: ACL 2023. pp. 4005–4019. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.findings-acl.247, https://aclanthology.org/2023.findings-acl.247
  * [5] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: Efficient Finetuning of Quantized LLMs. arXiv e-prints arXiv:2305.14314 (May 2023). https://doi.org/10.48550/arXiv.2305.14314
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [7] Feng, S.Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., Hovy, E.H.: A survey of data augmentation approaches for nlp. In: Findings (2021), https://api.semanticscholar.org/CorpusID:234093015
  * [8] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing (2020)
  * [9] Guan, T., Zan, H., Zhou, X., Xu, H., Zhang, K.: Cmeie: Construction and evaluation of chinese medical information extraction dataset. In: Natural Language Processing and Chinese Computing (2020), https://api.semanticscholar.org/CorpusID:222210416
  * [10] Guo, Z., Ni, Y., Wang, K., Zhu, W., Xie, G.: Global attention decoder for chinese spelling error correction. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 1419–1428 (2021)
  * [11] Hahn, U., Oleynik, M.: Medical information extraction in the age of deep learning. Yearbook of medical informatics 29(01), 208–220 (2020)
  * [12] He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. ArXiv abs/2110.04366 (2021)
  * [13] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning. pp. 2790–2799. PMLR (2019)
  * [14] Hughes, M., Li, I., Kotoulas, S., Suzumura, T.: Medical text classification using convolutional neural networks. In: Informatics for Health: Connected Citizen-Led Wellness and Population Health, pp. 246–250. IOS Press (2017)
  * [15] Jie, S., Deng, Z.: Convolutional bypasses are better vision transformer adapters. ArXiv abs/2207.07039 (2022)
  * [16] Kim, J., Kim, H.J., Cho, H., Jo, H., Lee, S.W., goo Lee, S., Yoo, K.M., Kim, T.: Ground-truth labels matter: A deeper look into input-label demonstrations. ArXiv abs/2205.12685 (2022), https://api.semanticscholar.org/CorpusID:249062718
  * [17] Kopiczko, D.J., Blankevoort, T., Asano, Y.M.: Vera: Vector-based random matrix adaptation. ArXiv abs/2310.11454 (2023), https://api.semanticscholar.org/CorpusID:264172315
  * [18] Le, T., Bertolini, M., No’e, F., Clevert, D.A.: Parameterized hypercomplex graph neural networks for graph classification. In: International Conference on Artificial Neural Networks (2021)
  * [19] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
  * [20] Levy, I., Bogin, B., Berant, J.: Diverse demonstrations improve in-context compositional generalization. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1401–1422. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-long.78, https://aclanthology.org/2023.acl-long.78
  * [21] Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) abs/2101.00190 (2021)
  * [22] Li, X., Lv, K., Yan, H., Lin, T., Zhu, W., Ni, Y., Xie, G.T., Wang, X., Qiu, X.: Unified demonstration retriever for in-context learning. ArXiv abs/2305.04320 (2023), https://api.semanticscholar.org/CorpusID:258557751
  * [23] Li, X., Qiu, X.: Finding support examples for in-context learning. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 6219–6235. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.findings-emnlp.411, https://aclanthology.org/2023.findings-emnlp.411
  * [24] Li, Y., Qian, B., Zhang, X., Liu, H.: Graph neural network-based diagnosis prediction. Big Data 8(5), 379–390 (2020)
  * [25] Li, Z., Chen, M., Yin, K., Tong, Y., Tan, C., Lang, Z., Tang, B.: Chip2022 shared task overview: Medical causal entity relationship extraction. In: Tang, B., Chen, Q., Lin, H., Wu, F., Liu, L., Hao, T., Wang, Y., Wang, H., Lei, J., Li, Z., Zong, H. (eds.) Health Information Processing. Evaluation Track Papers. pp. 51–56. Springer Nature Singapore, Singapore (2023)
  * [26] Liu, W., Tang, J., Qin, J., Xu, L., Li, Z., Liang, X.: Meddg: A large-scale medical consultation dataset for building medical dialogue system. ArXiv abs/2010.07497 (2020), https://api.semanticscholar.org/CorpusID:222377844
  * [27] Liu, X., Sun, T., Huang, X., Qiu, X.: Late prompt tuning: A late prompt could be better than many prompts. ArXiv abs/2210.11292 (2022)
  * [28] Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., Tang, J.: P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv abs/2110.07602 (2021)
  * [29] Liu, X., Ji, K., Fu, Y., Tam, W.L., Du, Z., Yang, Z., Tang, J.: P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In: Annual Meeting of the Association for Computational Linguistics (2022)
  * [30] Liu, Z., Qiao, A., Neiswanger, W., Wang, H., Tan, B., Tao, T., Li, J., Wang, Y., Sun, S., Pangarkar, O., Fan, R., Gu, Y., Miller, V., Zhuang, Y., He, G., Li, H., Koto, F., Tang, L., Ranjan, N., Shen, Z., Ren, X., Iriondo, R., Mu, C., Hu, Z., Schulze, M., Nakov, P., Baldwin, T., Xing, E.P.: LLM360: Towards Fully Transparent Open-Source LLMs. arXiv e-prints arXiv:2312.06550 (Dec 2023). https://doi.org/10.48550/arXiv.2312.06550
  * [31] Luo, X., Ni, Y., Tang, B.: Discussion on the application of text semantic matching technology in the field of chinese medical text from the perspective of competition. China Digital Medicine 11 (2021)
  * [32] Mahabadi, R.K., Henderson, J., Ruder, S.: Compacter: Efficient low-rank hypercomplex adapter layers. In: NeurIPS (2021)
  * [33] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., Zettlemoyer, L.: Rethinking the role of demonstrations: What makes in-context learning work? ArXiv abs/2202.12837 (2022), https://api.semanticscholar.org/CorpusID:247155069
  * [34] Moosavi, N.S., Delfosse, Q., Kersting, K., Gurevych, I.: Adaptable adapters. In: North American Chapter of the Association for Computational Linguistics (2022)
  * [35] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022)
  * [36] Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., Gurevych, I.: AdapterFusion: Non-destructive task composition for transfer learning. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pp. 487–503. Association for Computational Linguistics, Online (Apr 2021). https://doi.org/10.18653/v1/2021.eacl-main.39, https://aclanthology.org/2021.eacl-main.39
  * [37] Qin, C., Zhang, A., Dagar, A., Ye, W.: In-context learning with iterative demonstration selection. ArXiv abs/2310.09881 (2023), https://api.semanticscholar.org/CorpusID:264146526
  * [38] Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv e-prints arXiv:2302.06476 (Feb 2023). https://doi.org/10.48550/arXiv.2302.06476
  * [39] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [40] Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3, 333–389 (2009), https://api.semanticscholar.org/CorpusID:207178704
  * [41] Rubin, O., Herzig, J., Berant, J.: Learning to retrieve prompts for in-context learning. ArXiv abs/2112.08633 (2021), https://api.semanticscholar.org/CorpusID:245218561
  * [42] Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., Gurevych, I.: Adapterdrop: On the efficiency of adapters in transformers. In: Conference on Empirical Methods in Natural Language Processing (2020)
  * [43] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. Nature pp. 1–9 (2023)
  * [44] Sung, Y.L., Cho, J., Bansal, M.: Lst: Ladder side-tuning for parameter and memory efficient transfer learning. ArXiv abs/2206.06522 (2022)
  * [45] Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. ArXiv abs/1706.03762 (2017)
  * [46] Wang, X., Wei, J., Schuurmans, D., Le, Q., hsin Chi, E.H., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. ArXiv abs/2203.11171 (2022), https://api.semanticscholar.org/CorpusID:247595263
  * [47] Wang, X., Chen, L., Zhu, W., Ni, Y., Xie, G.T., Yang, D., Xiao, Y.: Multi-task entity linking with supervision from a taxonomy. Knowledge and Information Systems 65, 4335 – 4358 (2023), https://api.semanticscholar.org/CorpusID:258975891
  * [48] Wei, J., Wang, X., Schuurmans, D., Bosma, M., hsin Chi, E.H., Xia, F., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models. ArXiv abs/2201.11903 (2022), https://api.semanticscholar.org/CorpusID:246411621
  * [49] Wu, Z., Wang, S., Gu, J., Hou, R., Dong, Y., Vydiswaran, V.G.V., Ma, H.: Idpg: An instance-dependent prompt generation method. In: North American Chapter of the Association for Computational Linguistics (2022)
  * [50] Xiao, S., Liu, Z., Zhang, P., Muennighoff, N.: C-pack: Packaged resources to advance general chinese embedding (2023)
  * [51] Xiong, Y., Chen, M., Chen, Q., Tang, B.: Overview of the chip2021 shared task 1: Classifying positive and negative clinical findings in medical dialog. China Health Information Processing Conference (2021)
  * [52] Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, E., Zhang, Y.: A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. ArXiv abs/2312.02003 (2023), https://api.semanticscholar.org/CorpusID:265609409
  * [53] Zan, H., Li, W., Zhang, K., Ye, Y., Chang, B., Sui, Z.: Building a pediatric medical corpus: Word segmentation and named entity annotation. In: Chinese Lexical Semantics (2020), https://api.semanticscholar.org/CorpusID:236477750
  * [54] Zhang, J., Tan, M., Dai, P., Zhu, W.G.: Leco: Improving early exiting via learned exits and comparison-based exiting mechanism. In: Annual Meeting of the Association for Computational Linguistics (2023), https://api.semanticscholar.org/CorpusID:259370796
  * [55] Zhang, L., Zhang, L., Shi, S., Chu, X., Li, B.: Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. ArXiv abs/2308.03303 (2023), https://api.semanticscholar.org/CorpusID:260683267
  * [56] Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., Si, L., Ni, Y., Xie, G., Sui, Z., Chang, B., Zong, H., Yuan, Z., Li, L., Yan, J., Zan, H., Zhang, K., Tang, B., Chen, Q.: CBLUE: A Chinese biomedical language understanding evaluation benchmark. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7888–7915. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-long.544, https://aclanthology.org/2022.acl-long.544
  * [57] Zhang, X., Tan, M., Zhang, J., Zhu, W.: Nag-ner: a unified non-autoregressive generation framework for various ner tasks. In: Annual Meeting of the Association for Computational Linguistics (2023), https://api.semanticscholar.org/CorpusID:259370837
  * [58] Zhang, Y., Gao, X., Zhu, W., Wang, X.: Fastner: Speeding up inferences for named entity recognition tasks. In: International Conference on Advanced Data Mining and Applications (2023), https://api.semanticscholar.org/CorpusID:265214231
  * [59] Zhang, Y., Wang, P., Tan, M., Zhu, W.G.: Learned adapters are better than manually designed adapters. In: Annual Meeting of the Association for Computational Linguistics (2023), https://api.semanticscholar.org/CorpusID:259858833
  * [60] Zhang, Z., Zhu, W., Yan, J., Gao, P., Xie, G.: Automatic student network search for knowledge distillation. 2020 25th International Conference on Pattern Recognition (ICPR) pp. 2446–2453 (2021)
  * [61] Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.Y., Wen, J.R.: A Survey of Large Language Models. arXiv e-prints arXiv:2303.18223 (Mar 2023). https://doi.org/10.48550/arXiv.2303.18223
  * [62] Zheng, H., Zhu, W., Wang, P., Wang, X.: Candidate soups: Fusing candidate results improves translation quality for non-autoregressive translation. ArXiv abs/2301.11503 (2023), https://api.semanticscholar.org/CorpusID:256358677
  * [63] Zhou, B., Yang, G., Shi, Z., Ma, S.: Natural Language Processing for Smart Healthcare. arXiv e-prints arXiv:2110.15803 (Oct 2021). https://doi.org/10.48550/arXiv.2110.15803
  * [64] Zhou, X., Ni, Y., Xie, G.T., Zhu, W., Chen, C., Wang, T., Pan, Z.G.: Analysis of the health information needs of diabetics in china. Studies in health technology and informatics 264, 487–491 (2019), https://api.semanticscholar.org/CorpusID:201617388
  * [65] Zhu, W.: Autonlu: Architecture search for sentence and cross-sentence attention modeling with re-designed search space. In: Natural Language Processing and Chinese Computing (2021), https://api.semanticscholar.org/CorpusID:238862030
  * [66] Zhu, W.: MVP-BERT: Multi-vocab pre-training for Chinese BERT. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. pp. 260–269. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-srw.27, https://aclanthology.org/2021.acl-srw.27
  * [67] Zhu, W.: Mvp-bert: Multi-vocab pre-training for chinese bert. In: Annual Meeting of the Association for Computational Linguistics (2021), https://api.semanticscholar.org/CorpusID:237331564
  * [68] Zhu, W., He, Y., Chai, L., Fan, Y., Ni, Y., Xie, G.T., Wang, X.: paht_nlp @ mediqa 2021: Multi-grained query focused multi-answer summarization. In: Workshop on Biomedical Natural Language Processing (2021), https://api.semanticscholar.org/CorpusID:235097590
  * [69] Zhu, W., Li, W., Wang, X., Ji, W., Wu, Y., Chen, J., Chen, L., Tang, B.: Extracting decision trees from medical texts: An overview of the text2dt track in chip2022. In: Tang, B., Chen, Q., Lin, H., Wu, F., Liu, L., Hao, T., Wang, Y., Wang, H., Lei, J., Li, Z., Zong, H. (eds.) Health Information Processing. Evaluation Track Papers. pp. 89–102. Springer Nature Singapore, Singapore (2023)
  * [70] Zhu, W., Ni, Y., Wang, X., Xie, G.: Discovering better model architectures for medical query understanding. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers. pp. 230–237. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-industry.29, https://aclanthology.org/2021.naacl-industry.29
  * [71] Zhu, W., Tan, M.: SPT: Learning to selectively insert prompts for better prompt tuning. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 11862–11878. Association for Computational Linguistics, Singapore (Dec 2023), https://aclanthology.org/2023.emnlp-main.727
  * [72] Zhu, W., Wang, P., Ni, Y., Xie, G.T., Wang, X.: Badge: Speeding up bert inference after deployment via block-wise bypasses and divergence-based early exiting. In: Annual Meeting of the Association for Computational Linguistics (2023), https://api.semanticscholar.org/CorpusID:259370582
  * [73] Zhu, W., Wang, X.: Chatmed: A chinese medical large language model. https://github.com/michael-wzhu/ChatMed (2023)
  * [74] Zhu, W., Wang, X., Ni, Y., Xie, G.: Autotrans: Automating transformer design via reinforced architecture search. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds.) Natural Language Processing and Chinese Computing. pp. 169–182. Springer International Publishing, Cham (2021)
  * [75] Zhu, W., Wang, X., Zheng, H., Chen, M., Tang, B.: PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain. arXiv e-prints arXiv:2310.14151 (Oct 2023). https://doi.org/10.48550/arXiv.2310.14151
  * [76] Zong, H., Yang, J., Zhang, Z., Li, Z., Zhang, X.: Semantic categorization of chinese eligibility criteria in clinical trials using machine learning methods. BMC Medical Informatics and Decision Making 21 (2021), https://api.semanticscholar.org/CorpusID:233239370