# Good Data, Large Data, or No Data? Comparing Three Approaches in Developing
Research Aspect Classifiers for Biomedical Papers

Shreya Chandrasekhar, Chieh-Yang Huang, and Ting-Hao ‘Kenneth’ Huang
Pennsylvania State University, University Park, PA, USA
{sxc6175, chiehyang<EMAIL_ADDRESS>

###### Abstract

The rapid growth of scientific publications, particularly during the COVID-19
pandemic, emphasizes the need for tools to help researchers efficiently
comprehend the latest advancements. One essential part of understanding
scientific literature is research aspect classification, which categorizes
sentences in abstracts to Background, Purpose, Method, and Finding. In this
study, we investigate the impact of different datasets on model performance
for the crowd-annotated CODA-19 research aspect classification task.
Specifically, we explore the potential benefits of using the large,
automatically curated PubMed 200K RCT dataset and evaluate the effectiveness
of large language models (LLMs), such as LLaMA, GPT-3, ChatGPT, and GPT-4. Our
results indicate that using the PubMed 200K RCT dataset does not improve
performance for the CODA-19 task. We also observe that while GPT-4 performs
well, it does not outperform the SciBERT model fine-tuned on the CODA-19
dataset, emphasizing the importance of a dedicated and task-aligned datasets
dataset for the target task. Our code is available at
https://github.com/Crowd-AI-Lab/CODA-19-exp.

##  1 Introduction

The rapid growth of scientific publications, particularly during the COVID-19
pandemic, has made it increasingly challenging to keep up with the latest
research advancements. To address this issue, researchers have developed
various systems, such as search engines Lahav et al. (2022); Zhang et al.
(2020), visualization tools Hope et al. (2020); Tu et al. (2020), claim
verification systems Wadden et al. (2020); Pradeep et al. (2021), question-
answering systems Frisoni et al. (2022), and summarization techniques Meng et
al. (2021). These tools help efficiently comprehend publications by organizing
large amounts of information.

One critical aspect of understanding scientific literature is the
classification of sentences within abstracts according to their research
aspects, such as Background, Purpose, Method, and Finding. This is
particularly crucial in the biomedical domain, where abstracts tend to be
longer and more complex. With the annotated aspects, readers can quickly grasp
the key aspects of a scientific paper. For example, FacetSum Meng et al.
(2021) summarizes papers in four aspects to quickly convey the information.
Several datasets have been proposed for research aspect classification,
including PubMed 200K RCT Dernoncourt and Lee (2017), PubMed-PICO-Detection
Jin and Szolovits (2018), CODA-19 Huang et al. (2020), and more.

In this paper, we focus on the research aspect classification task using
crowd-annotated CODA-19 Huang et al. (2020) and explore the impact of
different datasets on model performance. Specifically, we investigate whether
the automatically curated large dataset (PubMed 200K RCT Dernoncourt and Lee
(2017)) can help the target task despite its shifted data distribution from
the target task. Additionally, we examine whether large language models
(LLMs), trained on a massive general textual corpus, can solve the task with
limited or no task-specific data provided. In particular, we evaluate six
LLMs, including three open-sourced models (LLaMA-65B Touvron et al. (2023),
MPT Team (2023), and Dolly-12B Databricks (2023)) and three closed models
(GPT-3 Brown et al. (2020), ChatGPT OpenAI (2022), and GPT-4 OpenAI (2023)).

Our study suggests that using PubMed 200K does not improve the performance of
the CODA-19 research aspect classification task, regardless of the approach
used. We experimented with (i) training purely on PubMed data, (ii) simply-
mixing PubMed and CODA-19 data, (iii) upsampling CODA-19 data and mixing it
with PubMed data to address the data imbalance issue, and (iv) using a two-
staged training approach where the model is trained on PubMed and CODA-19
sequentially. However, none of these approaches could improve the performance
of the target task. We hypothesize this is due to the use of the SciBERT
Beltagy et al. (2019), which has pre-trained on papers from scientific domains
and thus reduces the advantage of incorporating PubMed 200K. Our results also
showed that although GPT-4 performed well in both zero-shot and few-shot
settings, it was not able to outperform the SciBERT model fine-tuned on the
CODA-19 dataset. This finding suggests that having a dedicated dataset that
aligns well with the target task is still important.

##  2 Related Work

Moradi et al. (2021) compared BioBERT and GPT-3 in a few-shot learning setting
for sentence classification tasks and found that neither model could
outperform fully fine-tuned models. For biomedical information extraction
tasks, Jimenez Gutierrez et al. (2022) observed that GPT-3 could not
outperform the fine-tuned RoBERTa-large model, whereas Agrawal et al. (2022)
reported promising performance for GPT-3. Gururangan et al. (2020)
investigated whether adapting pre-trained models to the domain of the target
task could help, and concluded that domain-adaptive pretraining is always
helpful, highlighting the need for task-aligned data and training strategies.
In this paper, we focus on the aspect classification task on CODA-19 and
examine whether slightly domain-shifted large datasets and LLMs can improve
performance.

##  3 Methodology

In this section, we describe models trained to explore the impact of different
datasets and training strategies on the CODA-19 research aspect classification
task. For detailed model training details such as hyperparameters, please
refer to Appendix B.

###  3.1 Good Data: CODA-19

We use CODA-19 Huang et al. (2020) as our Good Data. CODA-19 consists of
clause-level aspect annotations for abstracts from medical papers, including
Background, Purpose, Method, Finding/Contribution, and Other. It contains
137K/15K/15K samples for train/validation/test sets. We fine-tune SciBERT
Beltagy et al. (2019) on the original CODA-19 to create SciBERTCODA and on the
position-encoded CODA-19 to create SciBERTCODA+Pos (see Section 3.4).

###  3.2 Large Data: PubMed

PubMed 200K Dernoncourt and Lee (2017) provides sentence-level research
aspects based on structured abstracts. It contains 2.2M/29K/29K samples for
the train/validation/test set. Despite including a larger amount of data, the
distribution is slightly different from the CODA-19 task. We first create two
models, SciBERTPubMed and SciBERTPubMed+Pos, by fine-tuning SciBERT on PubMed
200K. We also explore combining CODA-19 and PubMed with three strategies: (i)
simply-mixing the two datasets to create SciBERTMix+Pos+S; (ii) upsampling
CODA-19 ten times to balance the data size and mixing it with PubMed to create
SciBERTMix+Pos+U; and (iii) two-staged training, where the model is fine-tuned
on PubMed and CODA-19 sequentially to create SciBERTMix+Pos+T.

Setting | Background | Methods | Objective | Results | Conclusions | All
---|---|---|---|---|---|---
Support | 2,663 | 9,751 | 2,377 | 10,276 | 4,426 | 29,493
Model | FTR | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | ACC | F1
SciBERT | T |  .712 | .734 | .723 | .935 | .955 | .945 |  .787 |  .655 |  .715 | .918 | .932 | .925 | .858 | .848 | .853 | .887 | .832
SciBERT | T+P | .742 | .824 | .781 |  .945 | .970 |  .958 | .796 | .679 | .733 | .959 |  .940 |  .950 |  .946 | .946 | .946 | .919 | .873
bi-ANN+CRF | T | .707 |  .811 |  .756 | .955 |  .965 | .960 | .771 | .653 | .707 |  .956 | .948 | .952 | .946 |  .937 |  .942 |  .916 |  .863
Table 1: Performance on PubMed 200k Dernoncourt and Lee (2017), where best and second-best results are highlighted. The FTR column shows the feature used for the model, where T and T+P stand for text-only and position-encoded text. According to the overall accuracy and F1 score, our SciBERTPubMed+Pos performs the best. Setting | Background | Purpose | Method | Finding | Other | ALL
---|---|---|---|---|---|---
Support | 5,062 | 821 | 2,140 | 6,890 | 562 | 15,475
Data | Model | FTR | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | ACC | F1
C | SciBERT | T | .700 | .808 | .750 | .656 |  .588 | .620 | .716 | .635 | .673 | .802 | .743 | .772 | .797 | .867 | .830 | .746 | .729
C | SciBERT | T+P |  .825 | .794 | .809 | .638 | .655 | .647 | .741 | .665 | .701 | .823 |  .867 |  .845 | .819 | .843 |  .831 |  .803 | .767
P | SciBERT | T | .710 | .433 | .538 | .415 | .167 | .238 | .371 | .757 | .498 | .678 | .756 | .715 | - | - | - | .592 | .497
P | SciBERT | T+P | .854 | .415 | .559 | .233 | .251 | .241 | .362 |  .720 | .482 | .750 | .857 | .800 | - | - | - | .630 | .520
Mix${}_{\text{s}}$ | SciBERT | T+P | .762 |  .815 | .788 |  .674 | .446 | .537 | .677 | .640 | .658 | .824 | .843 | .833 | .808 | .642 | .716 | .777 | .706
Mix${}_{\text{u}}$ | SciBERT | T+P | .802 | .822 |  .812 | .669 | .585 | .624 | .734 | .636 | .681 |  .826 | .857 | .841 |  .812 | .820 | .816 | .799 | .755
Mix${}_{\text{t}}$ | SciBERT | T+P | .816 | .812 | .814 | .687 | .574 |  .625 |  .736 | .654 |  .693 | .827 | .870 | .848 | .807 |  .865 | .835 | .805 |  .763
C | BERT | T+P |  .846 | .761 | .801 | .626 |  .646 | .636 | .702 | .637 | .668 | .803 | .879 | .839 | .803 | .847 | .824 | .793 | .754
Mix${}_{\text{t}}$ | BERT | T+P | .828 | .775 | .801 | .663 | .639 | .651 | .715 | .639 | .675 | .808 |  .872 | .839 | .809 | .854 |  .831 | .795 | .759
Table 2: Performance on CODA-19 Huang et al. (2020), where best and second-
best results are highlighted. The Data column specifies the training data, C:
CODA-19, P: PubMed 200K, Mixs: simply-mixing, Mixu: upsampling, and Mixt: two-
staged training. The FTR column shows the feature used for the model, T: text-
only, and T+P: position-encoded text. According to the overall performance,
SciBERTCODA+Pos and SciBERTMix+Pos+T achieve the best performance.

###  3.3 No Data: LLMs

In the No-Data setting, the goal is to classify the research aspect of
abstract sentences using LLMs with limited or no task-specific training data.
LLMs are trained on a massive amount of web data, which has a different
distribution compared to our target dataset. To explore the performance of
LLMs in this scenario, we use zero-shot and few-shot classification with three
open-sourced models: LLaMA-65B Touvron et al. (2023), MPT-7B Team (2023), and
Dolly-12B Databricks (2023); and three closed models: GPT-3 Brown et al.
(2020), ChatGPT OpenAI (2022), and GPT-4 OpenAI (2023).

###  3.4 Position Encoding

To predict a research aspect for a sentence, the position of the sentence
within the whole abstract is important. Prior work such as Dernoncourt and Lee
(2017) used CRF to model the relationship between sentences. In this paper, we
incorporate position information by simply adding a position encoding to the
beginning of each sentence in the form of “[POSITION=0.38]”, where the number
represents the normalized sentence position, i.e., sentence_id /
#sentences_in_abstract. Examples of position-encoded data can be found in
Appendix A.

##  4 Experiments and Results

We conducted three experiments to (i) verify whether our fine-tuned SciBERT
model can outperform the model proposed in the original PubMed paper
Dernoncourt and Lee (2017); (ii) compare models trained on good data and large
data, aiming to examine whether a large automatically curated dataset can
enhance performance; (iii) benchmark the performance of open-sourced and
closed LLMs for the CODA-19 aspect classification task and compare them with
the best-performing SciBERT model.

###  4.1 Verifying the PubMed Model

In this experiment, we aim to assess the effectiveness of our fine-tuned
PubMed model by comparing it with the model reported in the original PubMed
paper Dernoncourt and Lee (2017).

#### Experimental Setup.

We evaluate two PubMed models in our study: SciBERTPubMed and
SciBERTPubMed+Pos. To compare with the reported model, we apply them to PubMed
200K test set, which contains 29,493 samples, to predict the PubMed label set
(Background, Methods, Objective, Results, and Conclusions). We report
precision, recall, and F1 scores for each label and calculate the accuracy and
macro F1 as overall metrics. Note that the micro F1 score provided in the
original PubMed paper is equivalent to accuracy since each instance is
assigned with only one label. To obtain the macro F1 score, we average the F1
scores across all labels.

#### Results.

The results shown in Table 1 demonstrate that our SciBERTPubMed+Pos model
outperforms the bi-ANN+CRF model Dernoncourt and Lee (2017) in both of
accuracy (0.919 vs. 0.916) and macro F1 score (0.873 vs. 0.863). These
findings suggest that, despite not considering a whole abstract
simultaneously, we can achieve competitive performance by incorporating
position encoding. Based on these results, we use SciBERTPubMed+Pos for
further comparisons.

Setting | Background | Purpose | Method | Finding | Other | ALL
---|---|---|---|---|---|---
Support | 250 | 250 | 250 | 250 | 250 | 1,250
Data | Model | FTR | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | ACC | F1
C | SciBERT | T+P | .492 | .784 | .605 | .879 | .696 | .777 | .710 | .684 | .697 | .669 | .632 | .650 | .983 | .696 | .815 | .698 | .709
P | SciBERT | T+P | .241 | .080 | .120 | .633 | .152 | .245 | .288 | .836 | .428 | .441 | .672 | .532 | - | - | - | .348 | .332
- | LLaMA | Zero | .212 | .160 | .182 | 1.000 | .012 | .024 | 1.000 | .004 | .008 | .700 | .056 | .104 | .180 | .748 | .291 | .196 | .122
- | Few | .402 | .556 | .466 | .800 | .240 | .369 | .663 | .228 | .339 | .484 | .596 | .534 | .556 | .968 | .707 | .518 | .483
- | MPT | Zero | .229 | .960 | .370 | .000 | .000 | .000 |  .923 | .048 | .091 | 1.000 | .004 | .008 | .746 | .564 | .642 | .315 | .222
- | Few | .230 | .304 | .262 | 1.000 | .008 | .016 | .667 | .008 | .016 | .289 |  .824 | .428 |  .748 | .604 | .668 | .350 | .278
- | Dolly | Zero | .208 |  .956 | .342 | .000 | .000 | .000 | .000 | .000 | .000 | .304 | .096 | .146 | .522 | .048 | .088 | .220 | .115
- | Few | .462 | .048 | .087 | .615 | .032 | .061 | .000 | .000 | .000 | .230 | .904 | .367 | .652 | .592 | .621 | .315 | .227
- | GPT-3 | Zero | .435 | .628 | .514 | .838 | .268 | .406 | .562 | .580 | .571 | .770 | .348 | .479 | .543 | .952 | .692 | .555 | .532
- | Few | .604 | .408 | .487 | .691 | .492 | .575 | .783 | .404 | .533 | .623 | .528 | .571 | .443 | .996 | .613 | .566 | .556
- | ChatGPT | Zero | .409 | .416 | .413 | .833 | .200 | .323 | .661 | .436 | .525 | .645 | .392 | .488 | .401 |  .992 | .571 | .487 | .464
- | Few | .446 | .516 | .479 | .833 | .200 | .323 | .663 | .464 | .546 | .621 | .472 | .536 | .461 | .988 | .628 | .528 | .502
- | GPT-4 | Zero |  .579 | .560 | .569 | .749 |  .548 |  .633 | .562 |  .692 | .620 |  .800 | .416 | .547 | .615 | .952 | .747 | .634 | .623
- | Few | .570 | .588 |  .579 |  .888 | .444 | .592 | .630 | .668 |  .649 | .679 | .600 |  .637 | .646 | .984 |  .780 |  .657 |  .647
Table 3: Performance on a randomly sampled subset of CODA-19 Huang et al.
(2020). We highlight the best and the second-best results. The Data column
specifies the training data: C: CODA-19 and P: PubMed. The FTR column shows
the feature used for the model, T: text-only, T+P: position-encoded text,
zero: zero-shot learning, and few: few-shot learning. Even the best-performing
LLM, GPT-4, does not outperform SciBERTCODA+Pos, showing the need for task-
aligned data. Open-sourced models (LLaMA, MPT, and Dolly) currently show lower
performance compared to closed models.

###  4.2 Good Data and Large Data

In this experiment, we aim to compare whether using a dataset with a larger
amount of samples but a slight domain shift could help improve the performance
of the CODA-19 aspect classification task.

#### Experimental Setup.

We evaluate seven different models on CODA-19 test set, which consists of
15,475 samples across five labels: Background, Purpose, Method, Finding, and
Other. The models include two trained on the CODA-19 dataset (SciBERTCODA and
SciBERTCODA+Pos), two trained on the PubMed dataset (SciBERTPubMed and
SciBERTPubMed+Pos), and three trained on a mix of the CODA-19 and PubMed
datasets (SciBERTMix+Pos+S, SciBERTMix+Pos+U, and SciBERTMix+Pos+T). For the
PubMed models, we map the predicted labels to the corresponding CODA-19 label
using a predefined mapping function: Background to Background, Methods to
Method, Objective to Purpose, Results to Finding, and Conclusions to Finding.
Note that Other (unclear or confusing sentences) in CODA-19 does not have a
corresponding label in PubMed. We report precision, recall, and F1 scores for
each label and calculate the accuracy and macro F1 as overall metrics. For the
PubMed models, the macro F1 score is averaged over the four valid labels.

#### Results.

The results of our experiment are presented in Table 2. We find that
SciBERTCODA+Pos and SciBERTMix+Pos+T achieve the highest accuracy (0.803 and
0.805) and macro F1 (0.767 and 0.763) scores, respectively, and outperform the
other models. The best performing PubMed model (SciBERTPubMed+Pos) does not
show any improvement over the performance on the CODA-19 test set (accuracy:
0.630 and macro F1: 0.520); and is particularly weak in identifying the
Purpose label (Purpose F1: 0.241). When comparing the different mixing
strategies, the models trained with simply-mixing and upsampling perform even
worse (accuracy: 0.777/0.799 and macro F1: 0.706/0.755). Although the two-
staged mixing strategy does not yield lower scores, it only achieves the same
results as SciBERTCODA+Pos.

Since SciBERT is pre-trained on a huge amount of scientific papers, with 82%
of the papers belonging to the biomedical domain Beltagy et al. (2019), we
also compare the performance of two approaches, the two-staged mixing
strategy, and pure fine-tuning, using BERT Devlin et al. (2019). This allows
us to eliminate the impact of SciBERT’s pretraining. The results, shown in the
last two rows of Table 2, indicate that the two-staged mixing strategy does
not yield the expected improvement (with overall F1 scores of 0.754 for
BERTCODA+Pos and 0.759 for BERTMix+Pos+T). Despite this, the overall score for
SciBERT remains higher. We hypothesize that the two-staged mixing strategy
with the classification objective may not be the best way for adapting the
model to a specific domain. Therefore, when large-scale pretraining, such as
training SciBERT, is not available, having a dedicated dataset that aligns
well with the target task is still important and will give the best
performance.

###  4.3 Comparison of No-Data

We investigate whether recent advances in LLMs can solve the CODA-19 aspect
classification task.

#### Experimental Setup.

Due to resource limitations, we experiment on a subset of CODA-19 test set,
where 250 samples for each of the five labels are randomly selected, resulting
in 1,250 samples.

We first include both SciBERTCODA+Pos and SciBERTPubMed+Pos for comparison and
report scores obtained by running on the evaluation set specifically for this
experiment. Additionally, we include six LLMs for comparison, i.e., LLaMA-65B,
MPT-7B, Dolly-12B, GPT-3, ChatGPT, and GPT-4, each in both the zero-shot and
few-shot settings. A total of 14 models are included. Note that out of the six
LLMs, LLaMA-65 is the only one not trained with instruction-following tasks.
We use the crowd workers’ annotation guidelines from CODA-19 Huang et al.
(2020) as our zero-shot prompt (see Table 7 in Appendix C for the actual
prompt). For the few-shot setting, we assume a scenario where users annotate a
single abstract as an example. Thus, we randomly select one abstract from
CODA-19 train set that contains four primary labels (Background, Purpose,
Method, and Finding). To avoid LLMs incorrectly considering the order of
samples as information, we shuffle the samples in the few-shot prompt (see
Table 8 in Appendix C for the actual prompt). To query each model, we use the
parameters described in Appendix B. Once we obtain the generated texts, we use
regex to parse the predicted label. When the predicted label is not in the
CODA-19 label sets or is missing, we treat it as Other. We report the per-
label precision, recall, and F1 score, as well as the overall accuracy and
macro F1 score.

#### Results.

The results of our experiment are presented in Table 3. SciBERTCODA+Pos
remains the best-performing model with an accuracy of 0.698 and a macro F1
score of 0.709. We observe that the zero-shot setting of LLaMA-65B performs
poorly, possibly due to the model not being trained for any instruction-
following tasks. The majority of its predictions are on Background and Other
labels, leading to very low recall for Purpose, Method, and Finding labels
(0.012, 0.004, and 0.056, respectively). Such biased prediction issues also
happen for MPT and Dolly in both zero-shot and few-shot settings even though
they are trained with instruction-following datasets, suggesting that there is
still a huge performance gap between open-sourced models and closed models.

When comparing closed models, our results show that ChatGPT performs worse
than GPT-3, possibly due to its optimization toward human-favored
conversation. On the other hand, GPT-4 outperforms GPT-3 by a large margin but
is still unable to outperform SciBERTCODA+Pos. While we believe that LLMs have
the potential to outperform SciBERTCODA+Pos in the future, our current results
emphasize the importance of having a dedicated dataset that aligns well with
the target task.

##  5 Conclusion

In this paper, we investigate the impact of different datasets and LLMs on the
CODA-19 research aspect classification task. Our findings show that using a
huge but slightly different dataset, PubMed 200K, does not help improve
performance. LLMs, trained with massive web corpus, are also unable to
outperform the SciBERT trained on the target dataset, emphasizing the
importance of task-aligned datasets. In the future, we will explore methods
for the model to consider the context and predict all sentences within a
single abstract at once.

## Acknowledgements

We thank all anonymous reviewers’ constructed feedback to improve this work.

## Limitations

One important aspect of achieving optimal performance when using LLMs is the
design of a high-quality prompt. In this study, we consider both zero-shot and
few-shot learning scenarios, which assume no or very limited task-specific
data. However, iteratively refining the prompt over time to obtain the best-
performing prompt may break the zero-shot or few-shot scenario. Moreover, the
final prompt used in this study is specifically designed to guide the crowd
workers in the annotation process of the CODA-19 dataset, with frequently
asked questions (FAQs) refined over time to address workers’ confusion. In a
real-world scenario, users would not have access to such helpful FAQs when
working on a new task. Therefore, the performance of LLMs may be lower in
practice.

Also, LLMs are susceptible to a data leakage problem due to their training
with Internet data. For example, ChatGPT is known to have been trained on
Internet data prior to September 2021. Considering that the CODA-19 dataset
was released in July 2020, with its train, validation, and test sets made
publicly available, there is a possibility that some closed models have seen
the exact test instances, leading to an unfair comparison. Since the training
data are not disclosed for the closed models, the impact of this exposure on
the models’ performance remains unknown.

## Ethics Statement

Deploying the model in this paper would likely result in unknown false
predictions. It requires further research to actually put it into practice.

## References

  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022\.  Large language models are few-shot clinical information extractors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1998–2022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _EMNLP_. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Databricks (2023) Databricks. 2023.  Databricks’ dolly, a large language model trained on the databricks machine learning platform.  https://github.com/databrickslabs/dolly.
  * Dernoncourt and Lee (2017) Franck Dernoncourt and Ji Young Lee. 2017.  Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts.  _arXiv preprint arXiv:1710.06071_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng et al. (2015) Lianyuan Feng, Shiyong Yao, Hejiang Sun, Nan Jiang, and Junjie Liu. 2015.  Tr-piv measurement of exhaled flow using a breathing thermal manikin.  _Building and environment_ , 94:683–693.
  * Frisoni et al. (2022) Giacomo Frisoni, Miki Mizutani, Gianluca Moro, and Lorenzo Valgimigli. 2022.  BioReader: a retrieval-enhanced text-to-text transformer for biomedical literature.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5770–5793, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Hope et al. (2020) Tom Hope, Jason Portenoy, Kishore Vasan, Jonathan Borchardt, Eric Horvitz, Daniel S Weld, Marti A Hearst, and Jevin West. 2020.  Scisight: Combining faceted navigation and research group detection for covid-19 exploratory scientific search.  _arXiv preprint arXiv:2005.12668_.
  * Huang et al. (2020) Ting-Hao’Kenneth’ Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and C Lee Giles. 2020.  Coda-19: Using a non-expert crowd to annotate research aspects on 10,000+ abstracts in the covid-19 open research dataset.  _arXiv preprint arXiv:2005.02367_.
  * Jimenez Gutierrez et al. (2022) Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022.  Thinking about GPT-3 in-context learning for biomedical IE? think again.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4497–4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Jin and Szolovits (2018) Di Jin and Peter Szolovits. 2018.  PICO element detection in medical text via long short-term memory neural networks.  In _Proceedings of the BioNLP 2018 workshop_ , pages 67–75, Melbourne, Australia. Association for Computational Linguistics.
  * Lahav et al. (2022) Dan Lahav, Jon Saad Falcon, Bailey Kuehl, Sophie Johnson, Sravanthi Parasa, Noam Shomron, Duen Horng Chau, Diyi Yang, Eric Horvitz, Daniel S Weld, et al. 2022\.  A search engine for discovery of scientific challenges and directions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11982–11990.
  * Meng et al. (2021) Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021.  Bringing structure into summaries: a faceted summarization dataset for long scientific documents.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1080–1089, Online. Association for Computational Linguistics.
  * Moradi et al. (2021) Milad Moradi, Kathrin Blagec, Florian Haberl, and Matthias Samwald. 2021.  Gpt-3 models are poor few-shot learners in the biomedical domain.  _arXiv preprint arXiv:2109.02555_.
  * OpenAI (2022) OpenAI. 2022.  ChatGPT: A large language model.  https://openai.com/blog/chatgpt/.  Knowledge cutoff: September 2021.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Pradeep et al. (2021) Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021.  Scientific claim verification with VerT5erini.  In _Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis_ , pages 94–103, online. Association for Computational Linguistics.
  * Team (2023) MosaicML NLP Team. 2023.  Introducing mpt-7b: A new standard for open-sourcely usable llms.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Tu et al. (2020) Jingxuan Tu, Marc Verhagen, Brent Cochran, and James Pustejovsky. 2020.  Exploration and discovery of the covid-19 literature through semantic visualization.  _arXiv preprint arXiv:2007.01800_.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zhang et al. (2020) Edwin Zhang, Nikhil Gupta, Rodrigo Nogueira, Kyunghyun Cho, and Jimmy Lin. 2020\.  Rapidly deploying a neural search engine for the COVID-19 Open Research Dataset.  In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_ , Online. Association for Computational Linguistics.

Label |  Id |  Sentence
---|---|---
Background | 0 |  [POSITION=0.00] Breathing is a high-risk behavior for spreading infectious diseases in enclosed environments ,
Background | 1 |  [POSITION=0.07] so it is important to investigate the characteristics of human exhalation flow and dispersal of exhaled air to reduce the risk .
Background | 2 |  [POSITION=0.14] This paper used two-dimensional time-resolved particle image velocimetry ( 2D TR-PIV ) to measure the exhaled flow from a breathing the rmal manikin .
Method | 3 |  [POSITION=0.21] Since the exhaled flow is transient and periodic ,
Method | 4 |  [POSITION=0.29] the phase-averaged method was used to analyze the flow characteristics .,
Table 4: A sample of position-encoded CODA-19 data extracted from the paper (Feng et al. (2015)). Here, Id indicates the sentence index with respect to the abstract. Removing the position encoding (e.g., [POSITION=0.00]), we could get the original CODA-19 data. Label |  Id |  Sentence
---|---|---
Methods | 3 |  [POSITION=0.38] The serum levels of follicle stimulating hormone (FSH), luteinizing hormone (LH), and estradiol (E(2)) were detected before and after the treatment.
Results | 4 |  [POSITION=0.50] After 12 weeks of treatment, HAMD scores in both groups decreased significantly (p<0.05) with no significant difference between the groups (p>0.05).
Results | 5 |  [POSITION=0.62] The levels of FSH decreased significantly and the level of E(2) increased significantly in both groups, and they changed more in the control group.
Results | 6 |  [POSITION=0.75] No side-effect of treatment was reported in either group during treatment.
Conclusions | 7 |  [POSITION=0.88] The Chinese medicinal formula GNL showed promise in relieving perimenopausal depression and merits further study.
Table 5: A sample of position-encoded PubMed data extracted from Paper ID:
19769482. Here, Id indicates the sentence index with respect to the abstract.
Removing the position encoding (e.g., [POSITION=0.38]), we could get the
original PubMed data.

##  Appendix A Sample Data

In this section, we show some sample data for CODA-19 dataset (Table 4) and
the PubMed dataset (see Table 5). As shown in the table, [POSITION=0.38] is
the position encoding we added to inject the positional information.

##  Appendix B Training and Testing Details

Here, we describe all the training details for the models we build in this
study. All the models are trained using PyTorch Paszke et al. (2019) and
HuggingFace Wolf et al. (2019).

  * •

SciBERTCODA. We fine-tune SciBERT using the original CODA-19 training set
using the hyperparameters listed in Table 6.

  * •

SciBERTCODA+Pos. The position encoding is first added to create the position-
encoded CODA-19 dataset. We then fine-tune SciBERT on the created dataset
using the hyperparameters listed in Table 6.

  * •

SciBERTPubMed. Similar to the above but on the original PubMed dataset.

  * •

SciBERTPubMed+Pos. Similar to the above but on the position-encoded PubMed
dataset.

  * •

SciBERTMix+Pos+S. We first turn the position-encoded PubMed’s label into the
CODA-19 label space using the pre-defined mapping: Background to Background,
Methods to Method, Objective to Purpose, Results to Finding, and Conclusions
to Finding. Second, we simply mix the position-encoded CODA-19 training set
with the position-encoded PubMed training set together to create a simply-
mixing dataset. We then fine-tune SciBERT on the created dataset using the
hyperparameters listed in Table 6. Note that the CODA-19 validation set is
used to checkpoint the best model.

  * •

SciBERTMix+Pos+U. As the data sizes of CODA-19 and PubMed differ a lot (137K
vs. 2.2M in the training set), we upsample position-encoded CODA-19’s training
set 10 times to create a more balanced upsampling dataset. Before mixing,
PubMed’s label has been transferred to the CODA-19 label space using the pre-
defined mapping function. We then fine-tune SciBERT using this dataset with
the pre-defined hyperparameters (Table 6). Again, the CODA-19 validation set
is used to checkpoint the best model.

  * •

SciBERTMix+Pos+T. For the two-staged training strategy using PubMed and
CODA-19 dataset, we first fine-tune SciBERT on the position-encoded PubMed
dataset with the pre-defined hyperparameters (Table 6). Here, the PubMed
validation set is used to checkpoint the best model. In the second stage, we
fine-tune the checkpointed model on the position-encoded CODA-19 dataset with
the pre-defined hyperparameters (Table 6).

  * •

LLaMA. We obtain LLaMA-65B from the official
Github111https://github.com/facebookresearch/llama and run the generation
using HuggingFace’s interface Wolf et al. (2019). Temperature sampling is used
for text generation with temperature = 0.1, num_beams = 1, top_p = 0.95,
repetition_penalty = 1.0, min_new_tokens = 1, and max_new_tokens = 10.

  * •

MPT. We use mosaicml/mpt-7b-instruct and run the generation using
HuggingFace’s interface Wolf et al. (2019) with the same parameters described
in LLaMA.

  * •

Dolly. We use databricks/dolly-v2-12b and run the generation using
HuggingFace’s interface Wolf et al. (2019) with the same parameters described
in LLaMA.

  * •

GPT-3. We use text-davinci-003 with the parameters: temperature = 0.0,
max_tokens = 10, top_p = 0.95, frequency_penalty = 0.0, and presence_penalty =
0.0.

  * •

ChatGPT. We use gpt-3.5-turbo with the same parameters described in GPT-3.
Note that when calling ChatGPT, we simply put all the prompts in a single user
input.

  * •

GPT-4. We use gpt-4 with the same parameters described in GPT-3. Again, when
calling GPT-4, we simply put all the prompts in a single user input.

Hyperparameter | Value
---|---
Model | scibert_scivocab_uncased
Batch Size | 64
Learning Rate | 1e-5
Epochs | 20
Metric for Best Model | Evaluation Accuracy
Max Sequence Length | 128
Warmup Ratio | 0.1
Early Stopping Patience | 6
Table 6: General hyperparameters used for training the models. We used
HuggingFace’s Trainer for fine-tuning all the models. Parameters not specified
here remain the default values.

##  Appendix C Prompts

Table 7 and Table 8 show the zero-shot prompt and the few-shot prompt we used
for querying LLMs.

Zero-shot Prompt
---
Classify the given text into one of the following labels.
[Background]: Text segments answer one or more of these questions: Why is this
problem important?, What relevant works have been created before?, What is
still missing in the previous works?, What are the high-level research
questions?, How might this help other research or researchers?
[Purpose]: Text segments answer one or more of these questions: What specific
things do the researchers want to do?, What specific knowledge do the
researchers want to gain?, What specific hypothesis do the researchers want to
test?
[Method]: Text segments answer one or more of these questions: How did the
researchers do the work or find what they sought?, What are the procedures and
steps of the research?
[Finding]: Text segments answer one or more of these questions: What did the
researchers find out?, Did the proposed methods work?, Did the thing behave as
the researchers expected?
[Other]: Text fragments that do NOT fit into any of the four categories above.
Text fragments that are NOT part of the article. Text fragments that are NOT
in English. Text fragments that contains ONLY reference marks (e.g.,
"[1,2,3,4,5") or ONLY dates (e.g., "April 20, 2008"). Captions for figures and
tables (e.g. "Figure 1: Experimental Result of …", or "Table 1: The Typical
Symptoms of …") Formatting errors. I really don’t know or I’m not sure.
FAQs
1\. This text fragment has terms that I don’t understand. What should I do?
Please use the context in the article to figure out the focus. You can look up
terms you don’t know if you feel like you need to understand them.
2\. This text fragment is too short to mean anything. What should I do? If the
text fragment is too short to have significant meanings, you could consider
the entire sentence and answer based on the entire sentence.
3\. This text fragment is NOT in English. What should I do? If the whole
fragment (or the majority of words in the fragment) is in Non-English, please
label it as "Other". If the majority of the words in this fragment are in
English with a few non-English words, please judge the label normally.
4\. I’m not sure if this should be a "background" or a "finding." How do I
tell? When a sentence occurs in the earlier part of an article, and it is
presented as a known fact or looks authoritative, it is often a "background"
information.
5\. Do "potential applications of the proposed work" count as "background" or
"purpose"? It should be "background." The "purpose" refers to specific things
the paper wants to achieve.
6\. If the article says it’s a "literature review" (e.g., "We reviewed the
literature" / "In this article, we review.." etc), would we classify those as
finding/contribution or purpose? Most parts of a literature review paper
should still be "background" or "purpose", and only the "insight" drew from a
set of prior works can be viewed as a "finding/contribution".
7\. What should I do with the case study on a patient? Typically, it has a
patient come in with a set of signs and symptoms in the ER, and then the
patient gets assessed and diagnosed. The patient is admitted to the hospital
ICU and tests are done and they may be diagnosed with something else. In such
cases, please label the interventions done by the medical staff (e.g., CT
scans, X-rays, and medications given) as "Method", and the patient’s final
result (e.g. the patient’s pneumonia resolved and he was released from the
hospital) as "Finding/Contribution".
Classify the following sentence into one of the label: Background, Purpose,
Method, Finding, and Other.
Text: “‘{Target-Sentence}”’
The answer label for Text is [
Table 7: Zero-shot prompt used when calling LLMs (LLaMA, MPT, Dolly, GPT-3,
ChatGPT, and GPT-4). The {Target-Sentence} will be replaced by the sentence we
would like to predict. Few-shot Prompt
---
Classify the given text into one of the following labels.
[Background]: Text segments answer one or more of these questions: Why is this
problem important?, What relevant works have
$\cdots$
$\cdots$ (Same as the zero-shot prompt that describe the definition of the
label and FAQs. Skip for space.)
$\cdots$
Classify the following sentence into one of the label: Background, Purpose,
Method, Finding, and Other.
Text: “‘With the features of extremely high selectivity and efficiency in
catalyzing almost all the chemical reactions in cells ,”’
Label: [Background]
Text: “‘enzymes play vitally important roles for the life of an organism and
hence have become frequent targets for drug design .”’
Label: [Background]
Text: “‘by which users can easily obtain their desired results .”’
Label: [Method]
Text: “‘An essential step in developing drugs by targeting enzymes is to
identify drug-enzyme interactions in cells .”’
Label: [Background]
Text: “‘a user-friendly web server was established ,”’
Label: [Method]
Text: “‘called “ iEzy-Drug , ” in which each drug compound was formulated by a
molecular fingerprint with 258 feature components ,”’
Label: [Method]
Text: “‘and the prediction engine was operated by the fuzzy K-nearest neighbor
algorithm .”’
Label: [Method]
Text: “‘Although some computational methods were developed in this regard
based on the knowledge of the three-dimensional structure of enzyme ,”’
Label: [Background]
Text: “‘Here , we reported a sequence-based predictor ,”’
Label: [Purpose]
Text: “‘Moreover , to maximize the convenience for the majority of
experimental scientists ,”’
Label: [Method]
Text: “‘It is both time-consuming and costly to do this purely by means of
experimental techniques alone .”’
Label: [Background]
Text: “‘The overall success rate achieved by iEzy-Drug via rigorous cross-
validations was about 91 % .”’
Label: [Finding]
Text: “‘unfortunately their usage is quite limited because threedimensional
structures of many enzymes are still unknown .”’
Label: [Background]
Text: “‘each enzyme by the Chou ’s pseudo amino acid composition generated via
incorporating sequential evolution information and physicochemical features
derived from its sequence ,”’
Label: [Method]
Text: “‘{Target-Sentence}”’
The answer label for Text is [
Table 8: Few-shot prompt used when calling LLMs (LLaMA, MPT, Dolly, GPT-3,
ChatGPT, and GPT-4). The {Target-Sentence} will be replaced by the sentence we
would like to predict. The skipped label description and FAQs are the same as
the zero-shot prompt (see Table 7). The few-shot samples are from one single
abstract to simulate the scenario where people annotate some data as a
reference.


# Data acquisition system for muon tracking in a muon scattering tomography
setup

Subhendu Das  Sridhar Tripathy  Jaydeep Datta  Sandip Sarkar  Nayana Majumdar
Supratik Mukhopadhyay

###### Abstract

We report here the development of a multi-channel DAQ system for muon tracking
in a muon scattering tomography setup. The salient features of the proposed
DAQ system are direct acquisition and processing of LVDS signals, 500MHz
sampling frequency and scalability. It consists of front-end electronics stage
built around NINO ASIC. The back-end electronics is configured with
Intel®/Altera® MAX®-10 FPGA development board which transmits data to the
storage following UART protocol. The proposed DAQ system has been tested for
its performance using a position sensitive glass RPC detector with two-
dimensional 8 $\times$ 8 readout strip configuration.

###### keywords:

Muon Scattering Tomography , NINO ASIC , Field Programmable Gate Array , FPGA-
based DAQ ,

††journal: Nuclear Instrumentation and Methods A

[inst1]organization=Saha Institute of Nuclear Physics, A CI of Homi Bhabha
National Institute, addressline=Sector I, AF Block, Bidhannagar, city=Kolkata,
postcode=700064, country=India

[inst2]organization=University of California, addressline=One Shields Ave,
Davis, city=California, postcode=95616, country=USA

[inst3]organization=Center for Frontiers in Nuclear Science, Department of
Physics and Astronomy, addressline=Stony Brook University, 100 Nicolls Road,
Stony Brook, city=New York, postcode=11794, country=USA

##  1 Introduction

Muon Scattering Tomography (MST) is a non-destructive evaluation technique
used for investigating the internal structure and constituent materials of a
large and static object by utilizing the principle of multiple Coulomb
scattering of cosmic ray muons. While passing through matter medium, the muons
can suffer scattering owing to their electromagnetic interaction with the
atomic nuclei present in the medium [1, 2, 3, 4, 5, 6]. The net deflection of
a muon from its original trajectory can be represented as a Gaussian
distribution with standard deviation dependent on the momentum of muon and
thickness of the object in terms of radiation length vis-a-vis its atomic
number and density [7, 8, 9]. Therefore, determination of scattering angle by
tracking the muon trajectory can be utilized to identify the material if the
muon momentum is known. The tracking of muon may be accomplished with a series
of position sensitive detectors placed along the direction of muon
propagation. The two-dimensional position information of the muon event
obtained from each of them can be used to reconstruct the trajectory of the
muon. The scattering angle eventually can be determined from the incident and
scattered trajectories reconstructed using positions of the muon events at the
tracking detectors placed respectively before and after their passage through
the object.

Gaseous detectors are frequently used as tracking devices for their excellent
position and timing resolutions. A few more advantages, like low cost and
relatively easy production of large area coverage add to their wide acceptance
in this area of application [10, 11]. To ensure precise measurement of
scattering angle, which can be as small as few mrad for low density materials,
like Aluminium, the position resolution of the detectors should be of the
order of few hundreds of $\mu m$ [6, 12]. Obviously, fairly high readout
granularity with finer strip width of the order of mm or less is an essential
requisite for this to achieve. Eventually, the higher granularity combined
with sufficiently large coverage of the tracking detectors calls for a large
number of readout channels. Therefore, a cost effective solution for the
readout electronics turns out extremely important in planning and designing of
such an application. For data acquisition of a setup with large number of
input channels, FPGA based systems offer the optimal solution for the backend
signal processing and control because of the availability of large number of
I/O, parallel operation, software controlled reconfigurability and cost
effectiveness.

We have aimed to build a prototype setup for material discrimination utilizing
the technique of MST with an objective of its application in inspection of
civil structures [13, 14]. In the initial phase, we plan to implement single-
gap Resistive Plate Chamber (RPC) as position sensitive tracking detector in
the setup for detection of muon. The RPC in particular has been opted for its
simple and robust design, easy construction using inexpensive materials, yet,
very efficient performance along with excellent position and timing
resolution. The design and choice of materials for fabricating the RPC have
been optimized with numerical simulation of the electrical properties of the
detector [15]. In the setup, two sets of RPC, each containing three of them,
will be commissioned above and below the inspection volume. The two-
dimensional position information of the muon events recorded by the RPCs in
each set will be used to reconstruct the trajectories and determine the
scattering angle subsequently. A schematic layout of the prototype MST setup
has been shown in figure 1. The design has been optimized with detailed
numerical modelling and simulation of its performance [6]. In future, the RPCs
may be replaced with new generation Micro-Patterrn Gaseous Detectors (MPGDs)
to achieve more precise position information and improved performance of the
MST setup.

Figure 1: Schematic layout of the prototype MST setup.

We have configured a multi-parameter Data AcQuisition (DAQ) system for
acquiring two-dimensional position information (X,Y) of muon events from the
RPCs and their storage for further processing. It comprises of a front-end
stage receiving the analog signals from the RPCs followed by a back-end stage
for acquisition of valid information and their transmission to a permanent
storage for subsequent data analysis. The scheme and a few preliminary test
results of the DAQ system have been reported by us in an earlier publication
[16]. In the current paper, we present a detailed report on its configuration
along with performance test done using a single-gap glass RPC prototype.

In the following section 2, a comprehensive description of the DAQ
configuration with front-end and back-end electronics has been furnished along
with their functionality. The performance of the DAQ system has been validated
with several measurements which can be found in section 3 along with the
results. Finally, the section 4 has presented the summary and conclusion of
the work.

##  2 DAQ Configuration

The direct acquisition from the readout channels of the RPC has been done by
Front-End Electronics (FEE), built around a low-power, ultra-fast, amplifier
discriminator ASIC, namely NINO, fabricated with 0.25 $\mu$m CMOS technology.
It was initially developed for the Multi-gap Resistive Plate Chambers (MRPCs)
in the Time-of-Flight (TOF) array of the ALICE experiment [17, 18]. The Low
Voltage Differential Signal (LVDS) output from the FEE has been transferred to
the Back-End Electronics (BEE), configured with Altera/Intel® MAX®-10 FPGA. It
is a low-cost, single chip with small form factor and programmable logic
device. As and when prompted by trigger, the BEE has acquired and saved the
valid LVDS signals in parallel and transmitted the data in serial manner to a
personal computer (PC) following Universal Asynchronous Receiver / Transmitter
(UART) protocol for permanent storage. The schematic diagram of the proposed
DAQ system has been illustrated in figure 2. The design and configuration of
FEE and BEE stages have been described in the following sections 2.1 and 2.2,
respectively.

Figure 2: Schematic diagram of the DAQ system.

###  2.1  Front-End Electronics (FEE)

The NINO ASIC has been built as a discrminator to produce an output by
measuring Time-Over-Threshold (TOT) of the input signal for slewing
correction. The TOT is actually a measure of time lapsed between the leading
and trailing edges of the input pulse when they surpass a specific threshold
level of charge. The ASIC has been designed on the basis of a current to
voltage converter with a common gate circuit configuration followed by four
cascaded amplifiers with low gain and high bandwidth. There is a slow feedback
circuit to supply current for keeping the input stages correctly biased. An
offset is added here to adjust the threshold level for the measurement of TOT.
Finally, a stretcher is used before the LVDS output driver in order to match
the width requirement foreseen for any readout system. The NINO takes
differential input and its circuit is differential throughout to achieve an
improved immunity to cross talk. The architecture of one of its readout
channels has been schematically presented in figure LABEL:fig:nino_channel.
The characteristic features of the NINO have been furnished below.

  * 1.

8 input channels of either polarity

  * 2.

Adjustable threshold level of 10–100fC

  * 3.

Fast amplification with peaking time $<$ 1ns and rms resolution 20ps

  * 4.

8 LVDS output channels with level difference 300mV and time jitter $<$ 25ps

  * 5.

Operate with input capacitance 30pF

  * 6.

Power consumption 40mW per channel

A readout board of dimension 200mm $\times$ 23mm, with a common threshold
control for all the channels has been designed with a single NINO by the TIFR
and INO Collaboration [19], as shown in figure LABEL:fig:nino_board. The board
accepts either of the positive and negative polarity signals which is
converted into the differential type and fed to the inputs of the NINO. The
voltage requirement for the operation of the said board is $\pm$ 4V. We have
utilised the same NINO-board as the FEE of the present DAQ system.

(a)

(b)

Figure 3: (a) NINO-channel architecture, (b) NINO-board designed by the INO
Collaboration [19].

###  2.2  Back-End Electronics (BEE)

The BEE stage has been configured using Altera®/Intel® MAX®-10 FPGA-based
development board. One of the salient features of the said board is that it
has dedicated I/O for direct acquisition of LVDS signals produced by the FEE.
This eliminates the necessity of conversion of LVDS signals to TTL type before
the BEE stage and thereby, LVDS signal purity is maintained. To map the LVDS
connections between the FEE and BEE boards, a custom-designed connector board
has been fabricated, as shown in figure 4, with required 100$\Omega$
termination. The important features of the said FPGA-board have been mentioned
below.

  * 1.

2000 Logic Elements (LEs)

  * 2.

108 embedded memory blocks (Kbits)

  * 3.

12 user flash memory (KBytes)

  * 4.

Single internal configuration memory

  * 5.

16 embedded 18 x 18 multipliers

  * 6.

Phase Lock Loop (PLL) with maximum clock frequency 500MHz

  * 7.

101 I/O pins

  * 8.

On-board clock frequency 50MHz

  * 9.

CH340G chip-based USB-UART converter

Figure 4: Altera®/Intel® MAX®-10 FPGA-board connected to connector boards.

The code for signal acquisition and data transfer by the MAX®-10 FPGA has been
developed on the VHSIC Hardware Description Language (VHDL) platform. A custom
IP (Intellectual Property) core consisting of four components, namely, digital
delay module, controller, FIFO memory and UART module with TX pin, has been
generated for this purpose. The flowchart of the IP core has been depicted in
figure LABEL:fig:FPGA_operation_a. The Intel® Quartus® Prime software has been
used to compile and upload the configuration code. It has been implemented in
each of the input channels. The on-board Phase Lock Loop (PLL) facilitates
generation of variable clock frequencies up to a maximum of 500MHz for
sampling. A clock frequency 500MHz, as generated by the PLL, has been used for
sampling data in the delay module and FIFO memory to achieve 2ns resolution
while the frequency 50MHz generated by the on-board clock has been used for
the controller and UART TX module. When prompted by a trigger, the controller
has produced 260ns wide window. A digital delay of 128ns has been added to the
LVDS signal received from NINO to ensure its position inside the trigger
window, as shown in figure LABEL:fig:FPGA_operation_b. The FIFO memory has
been used to store the entire data lying inside the trigger window. When
prompted by the controller module, the data have been transmitted to the
connected PC through the TX pin of the UART. The information acquired for each
channel for each trigger have been saved in the PC for further offline
analysis. A code based on Python programming language has been developed and
used to acquire data on the PC using COM port and analyse subsequently.

(a)

(b)

Figure 5: (a) Flowchart of the IP core in MAX®-10 FPGA, (b) Trigger window and
the NINO signals.

##  3 DAQ Performance

The proposed DAQ system has been tested for its performance to acquire the
signals produced by cosmic muons in a single-gap glass RPC prototype. The
experimental setup has been described in the next section 3.1. It has been
followed by the section 3.2 where the functioning of the DAQ system for
acquisition of RPC signal has been discussed along with the measurement of
efficiency of the detector for muon detection. In the next section 3.3, the
process of trigger validation using different physical setups and logic
conditions have been presented along with the results. The scalability of the
DAQ system has been tested by studying the muon event distribution detected by
the whole RPC prototype using different configuration of the DAQ system. This
has been discussed in section 3.4.

###  3.1  Experimental Setup

A prototype RPC made with 2mm $\times$ 30cm $\times$ 30cm float glass plates
as resistive electrodes and 2mm gas gap has been used for detecting cosmic
muons. A gas mixture of 95% Freon and 5% Isobutane has been circulated through
the detector. Two readout panels each consisting of eight copper strips of
width 3cm and separation 2mm have been used for recording two-dimensional
(X,Y) position information. The panels have been placed in orthogonal manner
to each other outside the glass electrodes and insulated by a layer of mylar.
The readout scheme of the RPC prototype has been illustrated in figure
LABEL:fig:detector_config_xy. It shows that two NINO-boards of the FEE stage
have been connected to two readout panels for receiving detector analog
signals produced by the passage of muons. The LVDS signals generated by the
pair of NINO-boards corresponding to the RPC signal have been transmitted to
an Altera®/Intel® MAX®-10 FPGA-board of the BEE stage via the custom designed
connector board. The signals have been stored in the BEE when prompted by the
muon trigger produced from the coincidence of the scintillators and the data
subsequently have been transmitted to the PC. Several plastic scintillators of
different dimensions have been used in the setup for testing the DAQ system
with different trigger conditions. The schematic of the experimental setup for
testing the RPC prototype using three scintillators (SCN1, SCN2, SCN3) as
trigger detectors has been illustrated in figure LABEL:fig:detector_config_z.
The SCN1 and SCN2 are two finger-shaped scintillators with length 35cm and
their widths are 3cm and 5cm, respectively. The third scintillator has an area
25cm $\times$ 35cm which has an overlap with the entire active area of the
RPC.

(a)

(b)

Figure 6: (a) Readout configuration, (b) Experimental setup.

###  3.2 Detector Signal Acquistion

Upon passage of a cosmic muon through the active gaseous medium of the RPC, an
avalanche of electron-ion pairs is produced from ionization of gaseous
molecules followed by multiplication of charged ion pairs due to presence of
high electric field across the volume. The movement of the charged ions
towards respective electrodes induces current signal on the readout strips in
the vicinity of the event. In the present setup, the NINO-board has produced
TOT pulses corresponding to the signals induced on the readout strips.
Subsequently, the MAX®-10 FPGA-board has acquired the LVDS signals transmitted
from the NINO-channels following the method described in the section 2.2. A
schematic of TOT measurement procedure by the FPGA-board has been illustrated
in figure 7. Using the 500MHz maximum clock frequency available from the on-
board PLL, it is capable of achieving 2ns resolution for acquisition. A
spectrum of 260ns (total 130bit) of the trigger window has been stored keeping
TOT signals from the NINO near the middle region by adding 128ns digital delay
to avoid data loss. Each bit of memory has represented the state (0 or 1) of
the signal for the time interval of 2ns. Counting the consecutive high states
of the signal, the TOT has been calculated by the FPGA board for the
respective channels of the NINO.

Figure 7: Schematic of TOT measurement.

To validate performance of the DAQ system, several measurements have been
carried out to study the response of a single strip of the glass RPC
prototype. The finger-shaped SCN1 has been aligned along a particular readout
strip while the paddle-shaped SCN3 has been placed below the RPC covering the
whole active area. The trigger has been generated from the two-fold
coincidence of SCN1 and SCN3 to ensure the detection of muon events by the
single readout strip. A comparative study has been made between a typical
analog signal from the readout strip captured in oscilloscope and the same
acquired through the present DAQ system comprising of NINO at the FEE and
MAX®-10 FPGA at the BEE stages respectively. The spectra have been depicted in
figure LABEL:fig:osc_signal and LABEL:fig:fpga_osc. The figure
LABEL:fig:time_hist_fpga has shown a typical distribution of the TOT outputs
of the readout strip for muon events as acquired by the present DAQ system.
The efficiency of the strip of muon detection with respect to the
scintillators has been determined by dividing the muon counts obtained from
the strip by the number of triggers generated. In figure LABEL:fig:efficiency,
the efficiency as measured by using standard electronics and the present DAQ
system has been depicted for different working voltage. It shows that the
efficiency determined by using the present DAQ system is less than the other
measurement by 7-8% at higher voltages. The possible reason may be due to the
use of a threshold 100fC used in the NINO-board for producing the TOT pulse
which has curtailed the valid pulses with smaller charge content. It has been
corroborated by larger difference in efficiency at lower operating voltage
which has reached to about 35% at the operating voltage 9.6kV.

(a)

(b)

(c)

(d)

Figure 8: (a) A typical signal from a readout strip on oscilloscope, (b)
Corresponding output from the present DAQ system, (c) TOT distribution
histogram for a readout strip, (d) Efficiency of a readout strip as measured
by oscilloscope and the present DAQ system.

###  3.3 Trigger Validation

The DAQ system has been tested and validated by acquiring RPC pulses with
different trigger conditions produced by different physical setup and logic of
three plastic scintillators (SCN1, SCN2 and SCN3) described earlier in section
3.1. Two different cases of physical setup of the scintillators have been
shown in figures LABEL:fig:test_trigger_a and LABEL:fig:test_trigger_b.

(a)

(b)

Figure 9: (a), (b) Schematics of two cases of physical setup of plastic
scintillators SCN1, SCN2 and SCN3

The signals from the RPC for the muon events have been acquired for a logic
condition SCN1 & SCN2 & SCN3 from both the cases which are shown in figures
LABEL:fig:hit_8x8_pos_1 and LABEL:fig:hit_8x8_pos_2. For the second setup
shown in figure LABEL:fig:test_trigger_b, the result of another trigger
condition (SCN1 + SCN2) & SCN3 has been illustrated in figure
LABEL:fig:hit_8x8_pos_3. The trigger condition in all the cases has been
generated when all the three scintillators have generated signals in a
coincidence window of 50ns. For each event, a weight factor proportional to
the induced charge as indicated by the measured TOT pulse width has been
assigned to each strip. In case of n number of strips have fired and produced
pulse width $w_{i}$, the weight factor assigned to each strip has been
calculated as follows.

$\displaystyle\frac{w_{i}}{\sum_{i=1}^{n}w_{i}}~{}\text{for {n} = 1, 2 and
3.}$ (1)

Thus, when a single strip has been hit, the weight factor assigned to the
strip was 1. The position of the event in terms of the strip has been
calculated by a weighted sum of the strips for both the readout planes (X,Y).
The two-dimensional histograms of the muon events for the given trigger
conditions have been illustrated in figure 10 in terms of readout strips.

(a)

(b)

(c)

Figure 10: (a), (b) 2D histogram of muon events with trigger condition SCN1 &
SCN2 & SCN3 for two setups, (c) 2D histogram of muon events with trigger
condition (SCN1 + SCN2) & SCN3 for the second setup.

###  3.4 DAQ Scalability

The DAQ system has been tested for its scalability which is an important
requirement for building up a tomography setup consisting of multiple muon
tracking detectors with larger readout granularity. The present 8 $\times$ 8
readout configuration of the RPC has been operated with different BEE
configurations where each of the two NINO-boards of two readout planes (X,Y)
has been connected to a MAX®-10 FPGA-board. The FPGA-boards have been
configured in master-master and master-slave configuration. The muon trigger
has been produced from the coincidence of paddle-shaped SCN3 and a similar one
placed above covering the entire active area of the RPC. The trigger has been
passed to both of the FPGA-boards simultaneously. The schematic diagram of the
experimental setup have been illustrated in figure LABEL:fig:Scalable_DAQ_MM
and LABEL:fig:Scalable_DAQ_MS respectively.

(a)

(b)

Figure 11: (a) Scalable data acquisition system with Master-master
configuration, (b) Master-slave configuration.

A typical muon event histogram obtained with the master-slave configuration
has been depicted in figure 12. Events with one, two and three strips hit have
been considered for the hit map reconstruction and the events with greater
than three strips hit, treated as streamers, have been excluded. This
highlights the scalability feature of the DAQ where a 8 $\times$ 8 system can
be easily scaled to a 16 $\times$ 16 system or more by adding additional units
with very small changes in the software code.

Figure 12: 2D muon event distribution with master-slave configuration of BEE

##  4 Summary & Conclusion

In the present work, we have presented the development of a multi-channel DAQ
system to be used for muon tracking using RPC in a MST setup. The FEE stage of
the proposed DAQ system has been built around NINO ASIC and the BEE stage has
been configured using MAX®-10 FPGA development board. The valid data has been
transmitted to an external PC for offline processing through UART. The DAQ
system has been tested on a glass RPC for its performance. It has been found
capable of direct acquisition of LVDS signals from the FEE stage. The
availability of 500MHz sampling frequency on the FPGA-board has offered a
timing resolution of $\pm$ 2ns in measuring the TOT pulse provided by the
NINO. This has been found fairly acceptable for our application where the main
focus is to produce the map of the muon event position. It will also matter a
little for a readout strips of 1cm width to be used in future. We have
deliberately used a MAX®-10 FPGA development board, in the back-end, instead
of a custom-made FPGA board. The custom-made FPGA board usually has a very
long development cycle and moreover, it is comparatively costly. In
comparison, the readily available development boards are much cheaper and
there is no need a long development cycle. Many advantages of a custom-made
boards can be achieved by using a modular structure with easy scalability
feature. We have demonstrated in section 3.4 that the proposed DAQ has a
modular structure that can be easily scaled to accommodate higher number of
channels with small modifications in the software code.

## Acknowledgements

We are thankful to Mr. Shaibal Saha and other members of our laboratory for
assistance and advice for the work. We acknowledge the help extended by the
TIFR and INO Collaboration in making NINO-boards and the procurement of FPGA-
boards. The author S. Das thanks the UGC, Govt. of India, for financial
support.

## References

  * [1] Pesente et al., First results on material identification and imaging with a large-volume muon tomography prototype, Nuclear Instruments and Methods A 604 (2009) 738.
  * [2] C. Morris, C. Alexander, J. Bacon, K. Borozdin, D. Clark, R. Chartrand, C. Espinoza, A. Fraser, M. Galassi, J. Green, et al., Tomographic imaging with cosmic ray muons, Science & Global Security 16 (2008) 37.
  * [3] D. F. Mahon et al., A prototype scintillating-fibre tracker for the cosmic-ray muon tomography of legacy nuclear waste containers, Nuclear Instruments and Methods A 732 (2013) 408.
  * [4] V. Anghel et al., A plastic scintillator-based muon tomography system with an integrated muon spectrometer, Nuclear Instruments and Meth ods A 798 (2015) 12.
  * [5] J. Burns, S. Quillin, M. Stapleton, C. Steer, S. Snow, A drift chamber tracking system for muon scattering tomography applications, Journal of Instrumentation 10 (2015) P10041.
  * [6] S. Tripathy, J. Datta, N. Majumdar, S. Mukhopadhyay, Material discrimination in cosmic muon imaging using pattern recognition method, Journal of Instrumentation 15 (2020) P06029.
  * [7] V. L. Highland, Some practical remarks on multiple scattering, Nuclear Instruments and Methods in Physics Research 129 (1975) 497.
  * [8] G. R. Lynch, O. I. Dahl, Approximations to multiple coulomb scattering, Nuclear Instruments and Methods in Physics Research B 58 (1991) 6.
  * [9] J. e. a. P. D. G. Beringer, A study of time over threshold (tot) technique for plastic scintillator counter, Physical Review D 86 (2012) 010001.
  * [10] P. Baesso, D. Cussans, C. Thomay, J. J. Velthuis, J. Burns, C. Steer, S. Quillin, A high resolution resistive plate chamber tracking system developed for cosmic ray muon tomography, Journal of Instrumentation 8 (2013) P08006.
  * [11] L. Shi, Y. Wang, X. Huang, X. Wang, W. Zhu, Y. Li, J. Cheng, A high time and spatial resolution mrpc designed for muon tomography, Nuclear Instruments and Methods in Physics Research A 652 (2011) 16.
  * [12] C. Gnanvo, L. V. Grasso III, M. Hohlmann, J. B. Locke, A. Quintero, D. Mitra, Imaging of high-z material for nuclear contraband detection with a minimal prototype of a muon tomography station based on gem detectors, Journal of Instrumentation 9 (2014) C12038.
  * [13] S. Tripathy, J. Datta, N. Majumdar, S. Mukhopadhyay, Numerical evaluation of a muon tomography system for imaging defects in concrete structures, The European Physical Journal Plus 136:824 (2021).
  * [14] S. Das, S. Tripathy, P. Jagga, P. Bhattacharya, N. Majumdar, S. Mukhopadhyay, Muography for civil inspection, Instruments 6 (2022) 77.
  * [15] S. Das, J. Datta, N. Majumdar, S. Mukhopadhyay, Studies on electrical properties of resistive plate chamber (rpc), Journal of Instrumentation 17 (2022) P09041.
  * [16] S. Tripathy, S. Das, J. Datta, N. Majumdar, S. Mukhopadhyay, Precise tracking of cosmic muons using the time-over-threshold property of nino asics, Journal of Instrumentation 15 (2020) C11013.
  * [17] F. Anghinolfi, P. Jarron, A. Martemiyanov, E. Usenko, H. Wenninger, M. Williams, A. Zichichi, Nino: an ultra-fast and low-power front-end amplifierdiscriminator asic designed for the multigap resistive plate chamber, Nuclear Instruments and Methods in Physics Research A 533 (2004) 183.
  * [18] F. Anghinolfi, P. Jarron, F. Krummenacher, E. Usenko, M. Williams, Nino: An ultrafast low-power front-end amplifier discriminator for the time-of-flight detector in the alice experiment, IEEE Transactions On Nuclear Science 51 (2004) 1974.
  * [19] P. K. Kaur, Pathaleswar, M. Saraf, B. Satyanarayana, R. Shinde, Development of fast, low power 8-channel amplifier-discriminator board for the rpcs, Springer Proceedings in Physics 203 (2018) 571.


# Particle identification with the Belle II calorimeter using machine learning

Abtin Narimani Charan  Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany
<EMAIL_ADDRESS>
###### Abstract

I present an application of a convolutional neural network (CNN) to separate
muons and pions in the Belle II electromagnetic calorimeter (ECL). The ECL is
designed to measure the energy deposited by charged and neutral particles. It
also provides important contributions to the particle identification (PID)
system. Identification of low-momenta muons and pions in the ECL is crucial if
they do not reach the outer muon detector. Track-seeded cluster energy images
provide the maximal possible information. The shape of the energy depositions
for muons and pions in the crystals around an extrapolated track at the
entering point of the ECL is used together with crystal positions in
$\theta-\phi$ plane and transverse momentum of the track to train a CNN. The
CNN exploits the difference between the dispersed energy depositions from pion
hadronic interactions and the more localized muon electromagnetic
interactions. Using simulation, the performance of the CNN algorithm is
compared with other PID methods at Belle II which are based on track-matched
clustering information. The results show that the CNN PID method improves
muon-pion separation in low momentum.

##  1 Introduction

The Belle II experiment at the SuperKEKB $e^{+}e^{-}$ collider belongs to the
next generation of $B$ factories and is the upgraded version of the Belle
experiment. Among the main goals of the Belle II experiment there are the
search of New Physics and the high precision measurements of the Standard
Model parameters in the flavour sector [1]. Belle II intends to collect data
samples with an integrated luminosity of 50 ab-1 at predominantly the
$\Upsilon(4S)$ resonance. SuperKEKB’s goal for instantaneous luminosity is 6
$\times$ 1035 cm-2 s-1, which is 40 times larger than that of KEKB. This
luminosity can be achieved due to a significant decrease in the beam sizes by
a factor of 20 at the interaction point based on the nano-beam collision
scheme and by doubling the beam currents in both rings [2, 3]. With the above-
mentioned improvements, Belle II encounters higher beam background level. The
physics reach of Belle II can be enhanced if a better muon-pion separation can
be achieved. A better muon-pion separation for low-momenta tracks is
advantageous for $B$ physics with leptonic $\tau$ decays, since this can
reduce the number of pions misidentified as muons at low momentum. It is
crucial to rely on the information in the ECL because low-momentum muons (p
$\lesssim$ 0.6 GeV/$c$) cannot reach the dedicated, outermost detector
$K^{0}_{L}$-and-muon detector (KLM).

##  2 The ECL

The ECL consists of 8736 thallium-doped caesium iodide CsI(Tl) crystals
projected to the vicinity of the interaction point. It is immersed in a 1.5 T
magnetic field and composed of a barrel and two endcaps with polar angle
coverage of 12.4∘ – 155.1∘. The dimension of each crystal is $\sim$ 5$\times$5
cm2 with a length of 30 cm corresponding to 16.2 $X_{0}$ (radiation length).
The barrel has 6624 crystals positioned in 46 rings of crystals distributed in
the $\theta$-plane and each ring consists of 144 crystals in the $\phi$-plane
[1, 3]. In this study, tracks are extrapolated into the ECL. Then, at the
entry point of the track into the ECL, a window of 7$\times$7 crystals with
the measured deposited energy therein is selected. Since the number of
crystals in the barrel in each $\theta$-plane is equal to 144, the 7$\times$7
crystal images are symmetrical. Typical patterns of energy depositions for
muons and pions are shown in figure 1. Muons and pions both deposit energy by
ionization in the matter. Additionally, pions can undergo hadronic
interactions. Therefore, crystal images of muons are generally more localized
than pions.

Figure 1: Patterns of energy depositions for muons (blue) and pions (red)
inside the ECL barrel.

##  3 Charged particle identification at Belle II

There are two charged PID methods available in the Belle II analysis software
framework [4]. The first method is standard PID which is based on a
combination of measurements from different sub-detectors. For each charged
particle hypothesis ($i$) in each PID system, a likelihood
$\mathcal{L}^{\text{PID}}_{i}$ is determined. It is a function of the
probability density function parameters for a given set of observables. These
likelihoods can be used to construct a combined likelihood ratio for a
particular sub-detector. A binary likelihood ratio for the ECL system between
muon and pion is defined as
$R^{\text{ECL}}_{\mu/\pi}=\frac{\mathcal{L}^{\text{ECL}}_{\mu}}{\mathcal{L}^{\text{ECL}}_{\mu}+\mathcal{L}^{\text{ECL}}_{\pi}}$
which is used as benchmark for comparison in the study, and is indicated as
default. The standard PID in the ECL ($\mathcal{L}^{\text{ECL}}$) defines a
univariate likelihood as a function of $E/p$ i.e., the ratio of reconstructed
cluster energy over the momentum. The $E/p$ distribution is not very powerful
for muon-pion separation specifically for low momentum tracks (figure 2). The
second method is based on boosted decision trees (BDTs) [5]. It uses the
shower-shape information in the ECL combined with likelihood information from
other sub-detectors to train BDTs.

Figure 2: The distribution of the ratio of cluster energy over momentum
($E/p$) for simulated single track candidates of muons (blue) and pions (red)
inside the ECL barrel for low (left) and high (right) momentum range.

##  4 Convolutional neural network (CNN)

A convolutional neural network (CNN) is a type of deep neural network which is
used to recognize visual patterns from pixel images [6]. In this study, two
CNNs are trained with 7$\times$7 pixel images of muons and pions together with
crystal positions and transverse momentum of the track in the laboratory frame
(each pixel is a $\sim$ 5$\times$5 cm2 crystal). Separate CNNs are trained for
positive and negative charged tracks. This is due to the geometry of the ECL
i.e. the direction of the crystals. Approximately 1 million single muon and
pion candidates are generated with a flat distribution of transverse momentum
between 0.2 and 1.0 GeV/$c$. Each track is first reconstructed in the inner
tracking detectors, and then extrapolated into the ECL with Geant4 [7].

###  4.1 Inputs, pre-processing, and training

There are two types of inputs for the CNN. One is the energy depositions in
the 7$\times$7 pixel images and the other is a set of inputs which are fed
after the convolutional layers that includes pT, $\theta_{\text{ID}}$, and
$\phi_{\text{ID}}$ of the extrapolated tracks. The $\theta_{\text{ID}}$ and
$\phi_{\text{ID}}$ represent integer numbers corresponding to the location of
the crystal in the ECL. The energies in the pixel images are given as they
are, i.e., without any scaling applied since they already have small values.
However, there are very large values in a few pixel images of pions which have
energies more than 1 GeV. This is due to pion inelastic interaction with
nuclei producing protons. These large values are replaced with 1 GeV. Since
there are very few of these pixels, this adjustment is negligible since it
affects the standard deviation and mean of the energy depositions by 0.09 %
and 0.005 %, respectively. A threshold value of 1 MeV on the energy
depositions in the pixels is applied, so that pixels below this threshold are
assigned zero energy. The pT is already in the range of 0.2 – 1.0 GeV/$c$,
therefore no scaling is applied. The $\theta_{\text{ID}}$ and
$\phi_{\text{ID}}$ are used as categorical variables which are implemented as
an embedding in the network. An embedding is a distributed representation for
categorical variables where each category is mapped to a distinct vector that
a neural network can learn during training. The number of simulated events is
identical for signal and background. The equal size of the sample is important
to avoid bias against a particular type of particle or charge.

The CNN includes two parts. In the first part, the 7$\times$7 pixel images of
muons and pions are used as inputs for a convolutional layer. During
convolution, a window of 3$\times$3, called kernel, goes through each image. A
padding and stride of (1, 1) is used for the images. The padding adds one
pixel with value zero on the edge of the image which is beneficial to capture
more information on the edges. The stride refers to the amount of kernel
movement over the image. The feature map is set to 64. The second part of the
training involves a feed-forward neural network (FNN). In this part, the
results from the convolutional layer must be flattened and added on top of pT,
$\theta_{\text{ID}}$, and $\phi_{\text{ID}}$. The number of neurons in the
first and second layer of FNN are 3295 and 128, respectively. The dropout
layer with a value of 0.1 is used between the first and second layer of the
FNN. An Adam optimizer is used during the training with a learning rate of
0.001. The loss used in this study is CrossEntropy which is suitable for
binary classification problems. The number of epochs is set to 100. The model
is saved based on the lowest validation loss value in the corresponding epoch.
A schematic of the network is shown in figure 3.

Figure 3: Neural network architecture. See text for details.

###  4.2 Performance

The performance of the CNN is evaluated on a test dataset with an equal number
of muons and pions. The test dataset is generated and reconstructed with the
exact same conditions as the training dataset. The CNN is compared with two
other methods (default and BDT). The performance of all methods for both
charged tracks in a range of transverse momentum (0.52 $\leq$ pT $<$ 0.76
GeV/$c$) are shown in figure 4. The $\mu$ efficiency is defined as the ratio
of the number of correctly identified muons over the total number of muons.
The $\pi$ fake rate is defined as the ratio of the number of pions identified
as muons over the total number of pions. The CNN method outperforms the other
methods with AUC (Area Under the Curve) scores of 0.836 and 0.841 for positive
and negative charged tracks, respectively.

Figure 4: Comparison of CNN, BDT, and default PID. The left and right plots
show positive and negative charged tracks, respectively. The value in front of
each method shows the area under the curve (AUC).

There are cases (mostly pions) that the software framework does not match a
track with a cluster in the ECL. Since the CNN method does not rely on
clustering, a comparison is made among all tracks and tracks with and without
matched cluster in the inference phase. The comparison is shown in figure 5 in
which the blue ROC curves, representing the CNN (all tracks), can be
considered as an average between the red and green curves, which represent
tracks with and without matched clusters, respectively. The large difference
between red and green ones is due to statistics. The green ROC curve is
roughly 2.8% and 3.5% of the test dataset in the pT range of 0.52 – 0.76
GeV/$c$ for positive and negative charged tracks, respectively.

Figure 5: Comparison of CNN performance for all tracks, tracks with and
without matched cluster. The left and right plots show positive and negative
charged tracks, respectively.

Due to the higher beam background level at Belle II, a higher minimal energy
threshold for crystals may reduce the pile-up from beam background. In order
to check the robustness of the CNN method against different levels of beam
background, different energy thresholds for crystals of 0, 1, 2, 5, and 8 MeV
are tested and the results are shown in figure 6 (zero means no threshold).
Results show the CNN is robust against the choice of different energy
thresholds.

Figure 6: Comparison of CNN performance for different thresholds. The left and
right plots show positive and negative charged tracks, respectively.

##  5 Summary and outlook

This study shows that by using patterns of energy depositions in the ECL,
muon-pion separation can be improved for low-momenta tracks. The pion fake
rate is 4.2% and 6.7% larger at a typical working point of 90% muon
identification efficiency for positive and negative charged tracks,
respectively. The CNN method is available in the Belle II analysis software
framework and will be integrated as part of the standard Belle II
reconstruction software. This study can be extended to include additional low-
level ECL crystal information, e.g., pulse-shape discrimination [8, 9] which
is useful to separate hadronic and electromagnetic interactions. In order to
validate the CNN method on data, clean samples of muons and pions are selected
using $e^{+}e^{-}\rightarrow\mu^{+}\mu^{-}\gamma$ and $D^{*\,+}\rightarrow
D^{0}\,[\rightarrow K^{+}\,\pi^{-}]\,\pi^{+}$, respectively. These results are
underway.

## References

## References

  * [1] Kou E et al. 2019 The Belle II Physics Book Prog. Theor. Exp. Phys. 2019 123C01
  * [2] Akai K, Furukawa K and Koiso H (SuperKEKB) 2018 SuperKEKB Collider Nucl. Instrum. Meth. A 907, 188–99
  * [3] Abe T et al. 2010 Belle II Technical Design Report 1011.0352
  * [4] Kuhr T, Pulvermacher C, Ritter M, Hauth T and Braun N 2019 The Belle II Core Software Comput. Softw. Big Sci. 3 1
  * [5] Milesi M, Tan J and Urquijo P 2020 Lepton identification in Belle II using observables from the electromagnetic calorimeter and precision trackers EPJ Web Conf. 245 06023
  * [6] Bishop C M 2006 Pattern Recognition and Machine Learning (Singapore: Springer Science+Business Media, LLC)
  * [7] Agostinelli S et al. (GEANT4) 2003 GEANT4–a simulation toolkit Nucl. Instrum. Meth. A 506 250–303
  * [8] Longo S and Roney J M 2018 Hadronic vs. electromagnetic pulse shape discrimination in CsI(Tl) for high energy physics experiments JINST 13 P03018
  * [9] Longo S et al. 2020 CsI(Tl) pulse shape discrimination with the Belle II electromagnetic calorimeter as a novel method to improve particle identification at electron-positron colliders Nucl. Instrum. Meth. A 982 164562