# Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument
Reasoning in Civil Procedures with GPT4

Dan Schumacher1 Anthony Rios2
1MS Data Analytics, College of Business
2Department of Information Systems and Cyber Security
The University of Texas at San Antonio
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we present our system for the SemEval Task 5, The Legal
Argument Reasoning Task in Civil Procedure Challenge. Legal argument reasoning
is an essential skill that all law students must master. Moreover, it is
important to develop natural language processing solutions that can reason
about a question given terse domain-specific contextual information. Our
system explores a prompt-based solution using GPT4 to reason over legal
arguments. We also evaluate an ensemble of prompting strategies, including
chain-of-thought reasoning and in-context learning. Overall, our system
results in a Macro F1 of .8095 on the validation dataset and .7315 (5th out of
21 teams) on the final test set. Code for this project is available at
https://github.com/danschumac1/CivilPromptReasoningGPT4.

##  1 Introduction

Mastering the reasoning behind legal arguments is a fundamental skill required
of all law students. In this study, we develop a novel approach for SemEval
Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge Held
and Habernal (2024). The SemEval Task released a dataset that was scraped from
The Glannon Guide To Civil Procedure, a textbook designed for law students.
Specifically, given case law, a question, and a potential answer to that
question, students must be able to reason over the contextual information
(case law) to determine if the question is correct or not.

There has been substantial research in developing NLP-based reasoning systems
Guha et al. (2024); Bongard et al. (2022); Chalkidis (2023); Blair-Stanek et
al. (2023); Kuppa et al. (2023); Yu et al. (2022). The methods can be
categorized into two major frameworks: fine-tuning and large language model-
based (LLM) solutions. For fine-tuning approaches, Bongard et al. (2022)
introduced an approach that fine-tunes LegalBERT Chalkidis et al. (2020) and
developed several methods for handling long text that does not fit within the
token limitations of LegalBERT. For LLM solutions, Chalkidis (2023) explored
the use of ChatGPT for solving legal exams. Guha et al. (2024) introduced a
more comprehensive evaluation benchmark geared to large language models
consisting of 162 tasks. Many of their experiments show that GPT4 is one of
the top performing approaches for legal reasoning across all language models,
while Flan-T5-XXL is the best open-source option. Although showing substantial
generalization is important, developing specific prompting strategies for
different reasoning tasks can substantially improve performance. Hence, this
paper adds to existing literature on the exploration of prompting approaches
in the legal domain.

For this work, we adapt several prompting-based strategies to develop an LLM-
based solution for the shared task. Specifically, we combine a retrieval
system with in-context learning and chain-of-thought reasoning. There are
several studies showcasing the utility of in-context learning Liu et al.
(2022a); Lu et al. (2022) and chain-of-thought reasoning Wei et al. (2022);
Sun et al. (2023); Yao et al. (2024). Moreover, there is work using both
human-curated and machine-generated reasons. In this work, we focus on human-
generated reasons for the training data, and machine-generated examples are
only used at test times when human expert annotations are not provided.
Furthermore, rather than providing a step-by-step reasoning approach, which
may not make sense in this context, our approach is more similar to single-
step rationales Brinner and Zarrieß (2023); Yasunaga et al. (2023), which
simply provides a single step of reasoning for why an answer is correct or
incorrect.

In summary, this paper makes the following contributions to our solution for
the SemEval 2024 Task 5 shared task:

  * •

We evaluate prompting strategies using GPT4 that combine several popular
ideas, including in-context learning and chain-of-thought reasoning.

  * •

In-context learning can be sensitive to the actual choice of examples,
particularly when only a few examples are provided. Hence, we also explore an
ensemble of prompt-based predictions to improve overall performance.

  * •

Finally, we provide a unique error analysis where we found limitations and
common error types generated by GPT4 using our prompting strategies. For
example, when a part of an answer candidate is correct, but the reasoning is
wrong, GPT4 is likely to generate a false positive (i.e., predict it is
correct instead of incorrect).

##  2 RELATED WORK

Overall, there are three major areas of legal NLP: legal question-answering
Khazaeli et al. (2021); Kien et al. (2020); Ryu et al. (2023); Martinez-Gil
(2023); Wang et al. (2023), judgment prediction Masala et al. (2021); Valvoda
et al. (2023); Juan et al. (2023), and corpus mining (e.g., summarization,
text classification, information extraction, and retrieval-related research)
Poudyal et al. (2020); Li et al. (2022); Vihikan et al. (2021); Zhang et al.
(2023); Limsopatham (2021); de Andrade and Becker (2023). There has also been
some broad methodology work that is aimed at working on various legal tasks in
general (e.g., LegalBERT Chalkidis et al. (2020)).

The SemEval task is most similar to question-answering related research. In
the domain of legal question-answering, recent research efforts are focused on
creating new systems, developing evaluation criteria, and compiling datasets,
considering the significant variation across different legal fields. Khazaeli
et al. (2021) introduced a commercial question-answering system for legal
inquiries, leveraging information retrieval techniques, sparse vector search,
embeddings, and a BERT-based re-ranking system, trained on both general and
legal domain data. Ryu et al. (2023) developed a novel evaluation method for
LLM-generated texts that assess their validity using retrieval-augmented
generation, showing improved alignment with legal experts’ assessments and
effectiveness in identifying factual errors. Wang et al. (2023) created the
Merger Agreement Understanding Dataset (MAUD), a unique, expert-annotated
dataset for legal text reading comprehension, highlighting promising model
performance and the need for further improvement in understanding complex
legal documents.

From a methodological point-of-view instead of a task-oriented view, recent
research efforts have concentrated on the advancement of NLP-based reasoning
systems, with a particular focus on applications within the legal domain Guha
et al. (2024); Bongard et al. (2022); Chalkidis (2023); Blair-Stanek et al.
(2023); Kuppa et al. (2023); Yu et al. (2022). These efforts can be broadly
classified into two distinct methodologies: fine-tuning approaches and those
leveraging large language models (LLMs). Within the fine-tuning paradigm,
Bongard et al. (2022) have proposed modifications to LegalBERT Chalkidis et
al. (2020) aimed at enhancing its ability to process texts that exceed the
model’s inherent token limitations. This approach is representative of a
broader trend towards tailoring pre-existing models to better suit specific
textual analysis tasks in the legal sector. Kien et al. (2020) developed a
retrieval-based model employing neural attentive text representation with
convolutional neural networks and attention mechanisms for accurately matching
legal questions to relevant articles, demonstrating superior performance on a
Vietnamese legal question dataset.

Conversely, the exploration of LLMs has also been wide covering new general
approaches for legal question answering and reasoning to new datasets and
benchmarks. Yu et al. (2023) investigated the impact of chain-of-thought
prompts and fine-tuning methods on legal reasoning tasks, specifically the
COLIEE entailment task, and found that prompts based on legal reasoning
techniques and few-shot learning with clustered training data significantly
enhance performance. Chalkidis (2023) demonstrates the potential of utilizing
models like ChatGPT for complex reasoning tasks, such as solving legal
examination questions. Building on this, Guha et al. (2024) have introduced a
comprehensive evaluation framework designed specifically for assessing the
capabilities of LLMs across a suite of 162 tasks. This benchmark aims to
provide a more nuanced understanding of the strengths and limitations of LLMs
in the context of legal reasoning. Additionally, while employed within a
distinct domain from legal reasoning, a comparable methodology in prompt
engineering—encompassing chain of thought prompting and in-context
prompting—is illustrated in Liu et al.’s recent work Liu et al. (2022b).
Overall, compared to the prior work that developed methods and datasets for
generating answers to questions, the SemEval task focuses on understanding
whether a provided answer candidate is valid given a specific context.

##  3 METHOD

Figure 1: Overview of our prompting strategy.

We provide a high-level overview of our approach in Figure 1. Overall, we
explore three major different prompting approaches: Zero-shot prompting, few-
shot prompting, and few-shot prompting with chain-of-thought-like reasoning.
Moreover, we explore an ensemble of multiple approaches. The methods are
described in the following subsections.

###  3.1 Few-Shot Retrieval Augmented Chain-of-Thought Prompting

We refer to our approach as “Few-Shot & CoT & RAG.” Each example within the
dataset contains an Introduction, Question, Answer Candidate, Analysis, and
Label. For new examples at test time, we only have access to the Introduction,
Question, and Answer Candidate. The Introduction consists of a general
background about a legal case. The Question is about the case, and the Answer
Candidate is an answer to the Question. It is important to note that the
Question could be in question form, where the answer directly answers what is
asked. However, it may function as a fill-in-the-blank exercise, where the
question presents an incomplete statement that the Answer Candidate is
expected to complete. The Analysis, which is only provided in the training and
development datasets, is a detailed expert-defined explanation for why the
Answer Candidate is or is not valid. The Label is a TRUE or FALSE value, where
TRUE means that the Answer Candidate correctly addresses the question given
the provided context. Likewise, FALSE means that the Answer Candidate is
incorrect.

As shown in Figure 1, our prompting strategy contains three main components, a
system prompt, the in-context examples, and the final test instance we will
classify as TRUE or FALSE. The system prompt describes what the large language
model (LLM) should do. In the Figure, it is shown that we also added explicit
information to limit the model from generating non-relevant information.

The in-context examples are provided to the LLM before the final text
instance. Intuitively, the goal is to provide some examples of the task we are
accomplishing to better ground the LLM to make better-generated responses.
This is the Retrieval Augmented aspect of our system (i.e., RAG). While any
random examples could be provided, we search for the most relevant examples
for each test instance. Formally, given an input instance $x_{i}$ consisting
of a concatenated Introduction, Question, and Answer triplet $w$, we retrieve
the most similar examples $\\{x_{1},\ldots,x_{\mathcal{N}(x_{i})}\\}$, where
$\mathcal{N}(x_{i})$ are the $k$ most similar examples to $x_{i}$. Each
question-answer pair is embedded using LegalBERT. The in-context examples all
come from the provided training dataset. Once retrieved, all of the relevant
information for the retrieved examples are used in the prompt (i.e., the
Introduction, Question, Answer Candidate, Analysis, and Label). Figure 1 shows
an example with two in-context examples.

Finally, the last component of the prompt is the text example. Basically, for
every example we wish to make a prediction for, we pass the Introduction,
Question, and Answer Candidate. The model will first generate the Analysis for
that example, then it will generate the Label.

###  3.2 Ensemble

Besides the method described in the previous section, we also explored
prompting variants to create an ensemble. We describe each of the additional
methods below (besides the Few-SHOT & CoT & RAG method described in the
previous subsection).

#### Zero-shot.

This approach does not use any in-context examples. We provide the system
prompt and the test example to the model directly to get the final prediction.

#### Zero-shot & CoT.

This method builds on the Zero-Shot approach by adding the CoT aspect to the
test example. Note that this is not available at test time, so the model
generates an Analysis section without previous examples.

#### Few-Shot.

The few-shot approach will use multiple in-context examples. However, unlike
our main method explained in the previous section, it uses the same in-context
examples for all test cases. The examples were chosen in an ad-hoc manner with
an emphasis on relatively short examples to limit the number of tokens to
reduce costs.

#### Few-Shot & CoT.

This builds on the Few-Shot method, where ad-hoc in-context examples are still
used, but we also provide the CoT reasoning (Analysis) section to the in-
context examples. The GPT4 model will generate the Analysis section for each
test example.

#### Few-Shot & RAG.

This also builds on the Few-Shot method, but instead of using ad-hoc examples,
it uses LegalBERT and cosine similarity to find relevant in-context examples
for each test case (as explained in the previous section).

Overall, there are a total of 4 models in our ensemble. Few-Shot and Few-Shot
& RAG were not used, but we evaluated them. The models in the ensemble were
chosen by checking combinations on the validation dataset. To make a
prediction, we use voting with a threshold (i.e., where the votes are
processed to generate the proportion of TRUE values).

###  3.3 Model Details

For all of our prompts, we use GPT-4-1106-preview with a temperature of .7.
The similarity metric used for finding relevant in-context examples is cosine
similarity. Moreover, we choose a total of 2 in-context examples, consisting
of 1 TRUE example and 1 FALSE example. We searched for the best threshold
after voting by calculating the proportion of TRUE vs. FALSE predictions,
ultimately choosing .5 as the threshold.

There were many cases where GPT4 does not return a final label in an easy-to-
process fashion (i.e., it does not end with a TRUE or FALSE). We explored
multiple approaches to parse the answer. Our ultimate strategy involves re-
submitting examples to GPT-4 that initially failed to produce a valid label,
explicitly indicating the absence of the label, and prompting the model to
generate the correct information.

##  4 RESULTS

Method | Macro F1 | Acc.
---|---|---
Baselines
RoBERTa | .5128 | .2286
Legal-BERT | .5575 | .2941
Zero-shot | .6681 | .7857
Zero-Shot & COT | .7162 | .7500
Few-shot | .6935 | .7738
Few-shot & COT | .6762 | .7262
Few-shot & RAG | .6898 | .7500
Few-Shot & COT & RAG (Ours) | .7306 | .7857
Ensemble (Ours) | .8095 | .8571

Table 1: Validation dataset results

In this section, we present our own results on the validation dataset as well
as the final results in the competition.

Baselines In our experiments, using the validation data, we compare our
approach (Few-SHOT & CoT & RAG) with the other approaches used in the
Ensemble. Moreover, we compare our system to both RoBERTa Liu et al. (2019)
and Legal-BERT. However, because both RoBERTa and Legal-BERT are limited to
512 tokens, we split the Introduction into $b$ pieces. $b$ is calculated by
subtracting the number of words in the question and answer from 512. We halve
that result to accommodate the fact that the number of tokens typically
exceeds the number of words. Next, we divide the number of words in the
Introduction by the previously calculated number to obtain the total number of
windows. Finally, all of the tokens in the Introduction are evenly split into
the windows. Each piece of the introduction is appended to the Question and
Answer pair independently to generate multiple predictions for each instance.
We then use voting to make a final prediction for the entire sequence. This
method is similar to what was explored in Bongard et al. (2022).

Validation Results. The validation performances are shown in Table 1. We
observe that the fine-tuned methods (e.g., RoBERTa), perform less effectively
compared to all methods utilizing GPT-4. Between the fine-tuned methods, we
find that Legal-BERT outperforms RoBERTa, which is expected given Legal-BERT
was fine-tuned on relevant corpora.

Next, between GPT4 methods (not including the ensemble), we find that
performance varies substantially between .6681 and .7162 for Macro F1. All
methods outperform Zero-Shot. However, Zero-Shot & COT achieved the best
performance across all baseline methods for Macro-F1. When we compare the
baseline approaches to our method (Few-Shot & COT & RAG), we find that the
method outperforms all variations. From an ablation standpoint, removing RAG
has the biggest performance drop (.7306 vs. .6762). Interestingly, we find
that removing CoT has the second largest drop in performance (.7306 vs.
.6935), and removing the in-context examples has the smallest drop in
performance (.7306 vs. .7162). Before the study, we expected removing the in-
context examples would result in the largest performance drop.

# | User | Macro F1 | Acc.
---|---|---|---
1 | zhaoxf4 | .8231 | .8673
2 | irene.benedetto | .7747 | .8265
3 | kbkrumov | .7728 | .8367
4 | qiaoxiaosong | .7644 | .8163
5 | UTSA-NLP | .7315 | .7959
6 | kubapok | .6971 | .7857
7 | samyak | .6599 | .7449
8 | hrandria | .6327 | .6939
9 | Yuan_Lu | .6000 | .6327
10 | PengShi | .5910 | .6735
11 | msiino | .5597 | .5714
12 | Hwan_Chang | .5556 | .5918
13 | kriti7 | .5511 | .6020
14 | woody | .5510 | .6633
15 | odysseas_aueb | .5143 | .6122
16 | Manvith_Prabhu | .4966 | .6224
17 | lhoorie | .4957 | .5000
18 | yms | .4827 | .7245
19 | U_201060 | .4503 | .6633
20 | langml | .4375 | .4490
21 | lena.held | .4269 | .7449

Table 2: Final Competition Results. Our submission is in bold font.

Competition Results. In Table 2, we report the final results of the
competition, achieving a Macro F1 of .7315. But, why do we see such a large
performance drop between the competition and validation results (.7315 vs.
.8095)? We hypothesize two major reasons. First, we realized that the
validation dataset contains many Introduction-Question pairs identical to the
ones in the training dataset. Despite the Answer Candidates differing across
the two datasets, the substantial overlap in Introduction-Question pairs may
lead to an overestimation of our model’s performance on the validation
dataset, rendering it potentially too optimistic when applied to entirely new
data. Second, we spent time over-optimizing the ensemble on the validation
dataset causing overfitting issues (e.g., checking thresholds, model
combinations, and more). By generating a better validation split, we may have
seen better generalization.

| predFalse | predTrue
---|---|---
actFalse | 57 | 9
actTrue | 3 | 15

Table 3: Confusion matrix for the validation dataset. actFalse and actTrue
stand for actual True and False values, respectively. predFalse and predTrue
stand for predicted False and predicted True.

Error Analysis Our method resulted in twelve mistakes on the validation
dataset: ten false positives and two false negatives. The confusion matrix is
shown in Figure 3. In Table 4 (See Appendix), we categorize each false
positive into one of four categories: “Incorrect reasoning,” “Shared the same
introduction and question pair,” “lots of similar language”, and “Other.” With
only two false negatives, there are few useful patterns to understand among
the errors. Hence, we only have a general FN category. However, from the false
positives, we make two major findings which we describe below.

The first was when the answer candidate had the correct answer but incorrect
reasoning. Here is a toy example demonstrating this pattern:

> Introduction: Carlos enjoys riding his skateboard in the skate park.
> Unfortunately, during one of his rides, he fell and split his head open.
>
> Question: Should Carlos go to the hospital?
>
> Answer Candidate: Yes, Carlos should go to the hospital because he likes to
> kick-flip so much.
>
> Analysis: While it is correct that Carlos should go to the hospital for
> medical attention after splitting his head open, the reasoning provided in
> the answer candidate is flawed. The decision to seek medical help should be
> based on the severity of the injury and the need for professional medical
> treatment, not on Carlos’s enjoyment of skateboard stunts.

Intuitively, part of the Answer Candidate is correct, i.e., Carlos should go
to the hospital, yet the reasoning that states why he should go to the
hospital is wrong.

The second point is that three of our ten false positives all shared the same
introduction and question pair. The introduction contained more extraneous
information than usual and was 128 words longer than the average. In these
instances, our model would analyze the answer candidate as correct but without
taking into account the particular case that was asked about. Below is a toy
example:

> Introduction: My dog Louisa loves to learn new tricks, go for walks, eat her
> dinner, then sleep through the night
>
> Question: What does Louisa like to do after dinner?
>
> Answer Candidates:
>
>   1. 1.
>
> Learn new tricks (wrong)
>
>   2. 2.
>
> Go for walks (wrong)
>
>   3. 3.
>
> Eat her dinner (wrong)
>
>   4. 4.
>
> Sleep through the night (correct)
>
>

In the example, three of the answers are incorrect, while “Sleep through the
night” is the correct example. When the Introduction is long, the actual
context of the question may be ignored, which can be interpreted as a needle
in a haystack issue. Developing better systems that provide direct “attention”
to relevant information may improve performance.

Finally, a third point that we believe caused our model to predict incorrect
answer candidates is similar language in both the introduction and question
compared to the answer candidate.

> Introduction: While making eggplant Parmesan for the first time in a buttery
> dutch over Paige burned herself on the stove
>
> Question:  How does Paige remember this incident.
>
> Answer Candidates: She remembers making eggplant Parmesan for the first time
> in a buttery dutch oven fondly.

##  5 CONCLUSION

In this paper, we described our approach for 2024 SemEval Task 4, The Legal
Argument Reasoning Task in Civil Procedures. Specifically, we introduced a
GPT4 prompting-based strategy that achieved 5th place in the competition out
of 21 participants. Overall, we find that combining in-context learning, where
we use a retrieval-based approach to find relevant examples, as well as in-
context learning improves model performance. Based on our experiments, there
are three natural areas for future research. First, the actual Analysis
section used for chain-of-thought reasoning does not match traditional methods
which use step-by-step reasoning. Hence, a logical next extension is to reword
(potentially with GPT4) the Analysis section to provide a step-by-step
explanation for an answer. Second, this work was limited to 2 in-context
examples to limit API costs and allow us to test other models in our initial
experiments. However, extending that to 10 or more examples can potentially
improve performance. Third, the current approach relies on a closed-source
model (GPT4). Exploring open-source models, particularly smaller open-source
models such as T5 Chung et al. (2022) and LLama2 Touvron et al. (2023), is
important to better understand the impact of pretraining data on performance.

## ACKNOWLEDGEMENTS

This material is based upon work supported by the National Science Foundation
(NSF) under Grant No. 2145357.

## References

  * Blair-Stanek et al. (2023) Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023.  Can gpt-3 perform statutory reasoning?  _arXiv preprint arXiv:2302.06100_.
  * Bongard et al. (2022) Leonard Bongard, Lena Held, and Ivan Habernal. 2022.  The legal argument reasoning task in civil procedure.  In _Proceedings of the Natural Legal Language Processing Workshop 2022_ , pages 194–207.
  * Brinner and Zarrieß (2023) Marc Brinner and Sina Zarrieß. 2023.  Model interpretability and rationale extraction by input mask optimization.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13722–13744.
  * Chalkidis (2023) Ilias Chalkidis. 2023.  Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark.  _arXiv preprint arXiv:2304.12202_.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  Legal-bert: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * de Andrade and Becker (2023) Leonardo de Andrade and Karin Becker. 2023.  Bb25hlegalsum: Leveraging bm25 and bert-based clustering for the summarization of legal documents.  In _Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing_ , pages 255–263.
  * Guha et al. (2024) Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2024.  Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Held and Habernal (2024) Lena Held and Ivan Habernal. 2024.  SemEval-2024 Task 5: Argument Reasoning in Civil Procedure.  In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_ , Mexico City, Mexico. Association for Computational Linguistics.
  * Juan et al. (2023) Yining Juan, Chung-Chi Chen, Hsin-Hsi Chen, and Daw-Wei Wang. 2023.  Custodiai: A system for predicting child custody outcomes.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 10–16.
  * Khazaeli et al. (2021) Soha Khazaeli, Janardhana Punuru, Chad Morris, Sanjay Sharma, Bert Staub, Michael Cole, Sunny Chiu-Webster, and Dhruv Sakalley. 2021.  A free format legal question answering system.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 107–113.
  * Kien et al. (2020) Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. 2020.  Answering legal questions by learning neural attentive text representation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 988–998.
  * Kuppa et al. (2023) Aditya Kuppa, Nikon Rasumov-Rahe, and Marc Voses. 2023.  Chain of reference prompting helps llm to think like a lawyer.  In _Generative AI+ Law Workshop_.
  * Li et al. (2022) Xiang Li, Jiaxun Gao, Diana Inkpen, and Wolfgang Alschner. 2022.  Detecting relevant differences between similar legal texts.  In _Proceedings of the Natural Legal Language Processing Workshop 2022_ , pages 256–264.
  * Limsopatham (2021) Nut Limsopatham. 2021.  Effectively leveraging bert for legal document classification.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 210–216.
  * Liu et al. (2022a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022a.  What makes good in-context examples for gpt-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114.
  * Liu et al. (2022b) Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yuhang Chen, Yanqing Zhao, Hao Yang, and Yanfei Jiang. 2022b.  Interpretable online log analysis using large language models with prompt strategies.  In _Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022)_ , Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2022) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  In _The Eleventh International Conference on Learning Representations_.
  * Martinez-Gil (2023) Jorge Martinez-Gil. 2023.  A survey on legal question–answering systems.  _Computer Science Review_ , 48:100552.
  * Masala et al. (2021) Mihai Masala, Radu Cristian Alexandru Iacob, Ana Sabina Uban, Marina Cidota, Horia Velicu, Traian Rebedea, and Marius Popescu. 2021.  jurbert: A romanian bert model for legal judgement prediction.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 86–94.
  * Poudyal et al. (2020) Prakash Poudyal, Jaromír Šavelka, Aagje Ieven, Marie Francine Moens, Teresa Goncalves, and Paulo Quaresma. 2020.  Echr: Legal corpus for argument mining.  In _Proceedings of the 7th Workshop on Argument Mining_ , pages 67–75.
  * Ryu et al. (2023) Cheol Ryu, Seolhwa Lee, Subeen Pang, Chanyeol Choi, Hojun Choi, Myeonggee Min, and Jy-Yong Sohn. 2023.  Retrieval-based evaluation for llms: A case study in korean legal qa.  In _Proceedings of the Natural Legal Language Processing Workshop 2023_ , pages 132–137.
  * Sun et al. (2023) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023.  Text classification via large language models.  _arXiv preprint arXiv:2305.08377_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Valvoda et al. (2023) Josef Valvoda, Ryan Cotterell, and Simone Teufel. 2023.  On the role of negative precedent in legal outcome prediction.  _Transactions of the Association for Computational Linguistics_ , 11:34–48.
  * Vihikan et al. (2021) Wayan Oger Vihikan, Meladel Mistica, Inbar Levy, Andrew Christie, and Timothy Baldwin. 2021.  Automatic resolution of domain name disputes.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 228–238.
  * Wang et al. (2023) Steven H Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. 2023.  Maud: An expert-annotated legal nlp dataset for merger agreement understanding.  _arXiv preprint arXiv:2301.00876_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Yasunaga et al. (2023) Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H Chi, and Denny Zhou. 2023.  Large language models as analogical reasoners.  _arXiv preprint arXiv:2310.01714_.
  * Yu et al. (2022) Fangyi Yu, Lee Quartey, and Frank Schilder. 2022.  Legal prompting: Teaching a language model to think like a lawyer.  _arXiv preprint arXiv:2212.01326_.
  * Yu et al. (2023) Fangyi Yu, Lee Quartey, and Frank Schilder. 2023.  Exploring the effectiveness of prompt engineering for legal reasoning tasks.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13582–13596.
  * Zhang et al. (2023) Ruizhe Zhang, Qingyao Ai, Yueyue Wu, Yixiao Ma, and Yiqun Liu. 2023.  Diverse legal case search.  _arXiv preprint arXiv:2301.12504_.

Type | Frequency |  Example
---|---|---
FP: Incorrect Reasoning | 16.67% (1/12) |  Carlos enjoys riding his skateboard in the skate park…
FP: Shared the same introduction and question pair. | 25% (3/12) |  My dog Louisa loves to learn new tricks…
FP: Lots of Similar Language | 8.33% (2/12) |  While making Eggplant Parmesean…
FP: Other | 25% (3/12) |  …
FN | 25% (3/12) |  …

Table 4: Manual categorization of error types. False positives are categorized
as either “Incorrect Reasoning,” “Shared the same introduction and question
pair,” “lots of similar language,” or “Other.” We use a single FN category
because of the lack of errors to analyze.

##  Appendix A Error Analysis

Table 4 reports the basic statistics for the error types we observed in our
error analysis.


# When can networks be inferred from observed groups?

Zachary P. Neal
Michigan State University<EMAIL_ADDRESS>
###### Abstract

Collecting network data directly from network members can be challenging. One
alternative involves inferring a network from observed groups, for example,
inferring a network of scientific collaboration from researchers’ observed
paper authorships. In this paper, I explore when an unobserved undirected
network of interest can accurately be inferred from observed groups. The
analysis uses simulations to experimentally manipulate the structure of the
unobserved network to be inferred, the number of groups observed, the extent
to which the observed groups correspond to cliques in the unobserved network,
and the method used to draw inferences. I find that when a small number of
groups are observed, an unobserved network can be accurately inferred using a
simple unweighted two-mode projection, provided that each group’s membership
closely corresponds to a clique in the unobserved network. In contrast, when a
large number of groups are observed, an unobserved network can be accurately
inferred using a statistical backbone extraction model, even if the groups’
memberships are mostly random. These findings offer guidance for researchers
seeking to indirectly measure a network of interest using observations of
groups.

NOTE: This is a post-print. The version of record can be found at: Neal, Z. P.
(2024). When can networks be inferred from observed groups? Network Science,
12, 189-200. https://doi.org/10.1017/nws.2024.6

##  1 Introduction

Collecting network data directly from network members, for example through
surveys or interviews, can be challenging due to the resource intensive nature
of the data collection and the risks to data quality from missingness,
reporting errors, and reactivity (adams, 2020; Marsden, 2011). These
challenges have led network researchers to seek alternate, indirect
measurement methods. When the network of interest is undirected, one common
alternative involves inferring an unobserved network from observed groups such
as club memberships or event participations, using a suitably-binarized
projection (e.g., Breiger, 1974; Newman, 2004; Mizruchi, 1996; Andris et al.,
2015; Schaefer et al., 2010). However, little is known about the circumstances
under which a network inferred from observed groups accurately captures the
unobserved network of interest. That is, when can networks be inferred from
observed groups?

To answer this question, I perform a series of experiments, varying the
structure of the unobserved network being inferred, characteristics of the
observed groups, and the method used to infer a network from the groups. When
a small number of groups are observed, an unobserved network can be accurately
inferred using a simple unweighted two-mode projection, provided that each
group’s membership closely corresponds to a clique in the unobserved network.
In contrast, when a large number of groups are observed, an unobserved network
can be accurately inferred using a statistical backbone extraction model, even
if the groups’ memberships are mostly random. These findings suggest that
networks can be inferred from observed groups, and offers guidance on when
such inferences are sufficiently accurate to be used when data cannot be
collected directly from network members.

The remainder of the paper is organized in four sections. In the background
section, I review the potential of inferences from observed groups as a
possible solution to challenges to directly collecting network data. In the
methods section, I describe an experiment designed to evaluate the accuracy of
a network inferred from observed groups. In the results section, I report the
accuracy of networks inferred under experimentally-varied conditions,
highlighting when such inferences are and are not accurate. Finally, in the
discussion section, I identify opportunities for future research and offer
recommendations for researchers wishing to infer networks from observed
groups.

##  2 Background

One common approach to collecting network data is to collect data directly
from the network’s members. For example, if we want to know who your friends
are, there is a strong intuitive appeal to simply asking you “Who are your
friends?” Although there are many variations, direct collection of network
data typically takes place via a survey or interview, which includes one or
more ‘name generator’ questions like the one above (adams, 2020; Marsden,
2011). However, the direct collection of network data comes with a number of
challenges: it can be resource-intensive (adams, 2020; Marsden, 2011), it is
subject to measurement error (Wang et al., 2012) and missingness (Kossinets,
2006), and it may be impossible when network members are too young (Neal,
2020a) or not human (Krause et al., 2009).

The severity of these challenges varies by context, and strategies exist for
overcoming them. However, these challenges have led network researchers to
look for indirect methods of collecting network data. Among the most widely
used approaches involves inferring an unobserved network from observed groups
(e.g., Newman, 2004; Mizruchi, 1996; Andris et al., 2015; Schaefer et al.,
2010). In this paper, I define a ‘group’ simply as a collection of individuals
whose structure is unspecified (e.g., a party’s list of attendees, but not who
talked to whom), and a ‘network’ as a structure among individuals (e.g., who
talks to whom; Wellman, 1988; Neal, 2023). A key advantage to this approach
over direct data collection is that group “affiliations are often observable
from a distance (e.g., government records, newspaper reports), without having
to have special access to the actors” (Borgatti and Halgin, 2011). I focus on
contexts where the number of observed or observable groups $G$ is at least as
large as the number of $N$ actors who might affiliate with those groups (i.e.,
where $G\geq N$). This often occurs in contexts where membership in many
groups can be discerned from archival data, or collected through field-based
observations conducted over an extended period.

Breiger (1974) provided the most well-known illustration of how a one-mode
network could be derived from two-mode data about individuals’ group
affiliations, inferring a network among 18 women from observations of their
attendance at 14 social events. This approach proposes that a network of
shared group affiliations (i.e., a bipartite or two-mode projection) provides
some information about the network connections among the groups’ members. It
relies on the logic that if two people belong to many of the same groups or
participate in many of the same events (what Feld (1981) called ‘foci’), then
they likely interact and have or will form ties.

Transforming two-mode data into one-mode data via projection necessarily
involves the loss of some information. Nonetheless, it has been used to
indirectly measure networks in a wide range of contexts, and in some fields
has become the de facto standard approach. Unobserved networks of scientific
collaboration are inferred from researchers’ observed paper authorships (e.g.,
Newman, 2004), unobserved networks of corporate executives are inferred from
their observed board memberships (e.g., Mizruchi, 1996), unobserved networks
of political alliance are inferred from lawmakers observed memberships in
voting blocs (e.g., Andris et al., 2015), and unobserved social networks are
inferred from young childrens’ play groups (e.g., Schaefer et al., 2010).
However, despite its widespread use, it remains unknown whether or when a
network inferred from observed groups is accurate.

But, what does it mean to accurately infer a network from observed groups?
Figure 1 illustrates the relationship between an unobserved network, observed
groups, and inferred network (Peel et al., 2022). On the left is a one-mode
network of interest depicting the connections between seven agents (e.g.,
people). Although this network exists, we can not directly observe it, perhaps
because these agents declined to complete a network survey. Instead, we can
only observe these agents’ memberships in groups (e.g. attending events
together, belonging to the same club, etc.), which are driven at least in part
by their unobserved network ties (Feld, 1981; Schaefer et al., 2022; Neal,
2023). The example in Figure 1 illustrates four different observed groups.
These observed groups are simply sets of agents observed together (e.g., the
top left group includes the purple, red, and green agents), but do not contain
any information about the structure among their members. Notably, in some
cases, group membership corresponds to a clique in the unobserved network
(e.g., the top left group), while in other cases it does not (e.g., the top
right group). These observed groups can be summarized in a two-mode network in
which agents are connected to groups, and via projection, we can transform
this two-mode agents-to-groups network into a one-mode agents-to-agents
network using a projection. Accuracy in this context refers to the extent to
which the one-mode network obtained via this process is similar to the
unobserved network of interest. The goal, as Peel et al. (2022) explain, is to
accurately infer or ‘reconstruct’ a network of interest from indirect data,
here of observed groups. In the supplementary materials I explore the related
goal of inferring key characteristics of the network of interest, without
actually reconstructing the network itself.

Figure 1: Relationship between an unobserved network, observed groups, and
inferred network. Accuracy may depend on (A) the structure of the unobserved
network, (B) the number of observed groups, (C) the extent to which observed
group correspond to cliques in the unobserved network, and (D) the method used
to infer network relationships from group memberships.

At least four factors might influence how accurately an unobserved network can
be inferred from observed groups: (A) the structure of the unobserved network,
(B) the number of observed groups, (C) the extent to which observed groups
correspond to cliques in the unobserved network, and (D) the method used to
infer network relationships from group memberships. First, networks with
certain structures may be easier to accurately infer than others. For example,
because the projection of any bipartite network, “even a random bipartite
network…will be highly clustered” (Watts, 2008, p. 128), it may be easier to
accurately infer an unobserved one-mode network whose structure is clustered.
Second, inferences may be more accurate when they are based on a large number
of observed groups because such inferences can draw on more information.
Third, inferences may be more accurate when they are based on observed groups
whose membership closely corresponds to cliques in the unobserved network
(Guillaume and Latapy, 2004). Finally, the accuracy of inferences may depend
on how network edges are inferred from observed group memberships.

##  3 Methods

To understand when networks can be inferred from observed groups, I perform a
series of experiments that follow Figure 1. First, I generate a one-mode
network using one of five network models, or choose one of five one-mode
empirical networks, that serves as the hypothetical unobserved network of
interest (see 3.1). Second, I simulate the groups that a researcher might
observe as a result of this network, varying both their number and
correspondence to cliques in the unobserved network (see 3.2). Third, I infer
a one-mode network from these observed groups, using either a simple
unweighted two-mode projection or a backbone extracted using the stochastic
degree sequence model (see 3.3). Finally, I compute the similarity of the
unobserved network of interest and the inferred network (see 3.4).

###  3.1 Unobserved networks

When attempting to infer a unobserved network from observed groups, the
structure of the unobserved network is unknown, but may nonetheless impact the
accuracy of the inferences. Therefore, I explore the accuracy of inferences
when the unobserved network has a range of structures, using both artificial
and empirical networks.

Artificial networks generated using well-known network models are useful
because they have well-known structural properties. Here, I consider five such
models. First, I generate 50-node random networks using the Erdős-Rényi model,
where the probability of an edge is 0.08 (Erdős and Rényi, 1959). Second, I
generate 50-node small-world networks using the Watts-Strogatz model, where
each node in a ring lattice is initially connected to its four nearest
neighbors, then edges are re-wired with probability 0.05 (Watts and Strogatz,
1998). Third, I generate 50-node scale-free networks using the preferential
attachment model, where two edges are added in each step (Barabási and Albert,
1999). Fourth, I generate 50-node caveman networks that contain 10 cliques of
5-nodes each (Watts, 1999). Finally, I generate 50-node core-periphery
networks in which 10 nodes form a dense core ($d=0.85$), and 40 nodes in the
periphery are connected to 1 or 2 core nodes (Borgatti and Everett, 2000).
These specifications all yield networks containing 50 nodes and about 100
edges, and therefore hold network size and density constant.

Although these artificial network models are well understood, they can
generate networks that may not resemble real networks. Therefore, I also
consider five empirical networks: the interactions of 62 Dolphins in Doubtful
Sound (Lusseau et al., 2003), the marital network among 15 families in 15th
century Florence (Padgett and Ansell, 1993), the social relationships among 34
members of a Karate club (Zachary, 1977), the friendships among 71 lawyers in
a Northeastern US corporate law firm (Lazega, 2001), and the friendships among
32 workers in a tailor shop in Zambia (Kapferer, 1972). Although these
empirical networks vary slightly in size and density, they are sufficiently
similar to the five artificial networks to permit comparisons.

###  3.2 Observed groups

When attempting to infer a unobserved network from observed groups,
characteristics of the observed groups may impact the accuracy of the
inferences. If every observed group corresponds to a clique in the unobserved
network, and a sufficiently large number of groups are observed, then the
unobserved network can be inferred with perfect accuracy using a two-mode
projection of the observed groups (Guillaume and Latapy, 2004). This is
closely related to the NP-hard ‘clique cover problem,’ which involves finding
the smallest number of cliques that completely cover a network (Karp, 1972).
Although an unobserved network can be accurately inferred under these
conditions, in practice a researcher may only be able to observe a limited
number of groups, or may only observe groups that do not perfectly correspond
to cliques in the unobserved network. Therefore, I experimentally vary both
the number of observed groups, and the extent to which group memberships
correspond to cliques in the unobserved network.

First, given an unobserved network of $N$ nodes, I consider the accuracy of a
network inferred from observations of $N$, $2N$, $5N$, $10N$, $20N$, or $50N$
groups. Inferences drawn from more observed groups should be more accurate
because they are based on more information. The lower end of this experimental
range (i.e., observing $N$ groups) represents the fewest number of observed
groups from which any network structure could, in principle, be inferred
(Neal, 2012). The upper end of this experimental range (i.e., observing $50N$
groups) represents the largest number of groups that might typically be
observable. For example, Neal (2020b) inferred a network among 100 U.S.
Senators from an average of 3500 bill sponsorships (i.e., $35N$ groups).

Second, I consider the accuracy of a network inferred from observed groups
whose members have a 50%, 60%, 70%, 80%, 90%, or 100% chance of being members
of the same clique in the unobserved network. Inferences drawn from observed
groups that more closely correspond to cliques in the unobserved network
should be more accurate because they contain more information, and less noise,
about the structure of the network. The upper end of this experimental range
(i.e., 100%) represents a scenario in which each observed group’s membership
is simply a clique in the unobserved network. For example, a tightly-knit
clique of friends may be observed hanging out (i.e. an observed group) with no
one else present. The lower end of this experimental range (i.e., 50%)
represents a scenario in which members of observed groups may or may not be
members of the same clique in the unobserved network. For example, a group of
researchers may be observed writing a grant together (i.e., an observed
group), but only some of them are collaborators (i.e., they are not a clique).
In the supplementary materials I also consider one case outside these
experimental conditions, where a very large number of groups are observed
($200N-1000N$), but members of those groups are highly unlikely to be members
of a clique ($p=0.1$).

I use a model of team formation (Guimera et al., 2005) that has been
formalized as a two-mode generative model (Neal, 2023) to simulate the
memberships of groups that might be observed. The model first randomly chooses
a clique from the unobserved network. Given a clique containing $k$ nodes, it
then generates an observed group of $k$ members by filling each position with
either a member of the clique (with probability $p$) or someone else (with
probability $1-p$). When $p=1$, the observed group’s members are simply the
clique’s members. In contrast, when $p=0.5$, the observed group’s members may
or may not be the clique’s members.

This approach involves the analysis of simulated groups that a researcher
might observe, as opposed to actual groups that a researcher did observe.
However, this generative model has previously been shown to generate simulates
group that have characteristics of empirically-observed groups (Neal, 2023).
Additionally, using a generative model offers an important advantage over
using empirical data: it is possible to experimentally manipulate how many
groups are observed, and the extent to which those groups correspond to
cliques, and therefore to investigate the hypothesized role that these two
factors play in the accuracy of inferred networks.

###  3.3 Inferring a network

Given a set of observed groups organized as a two-mode network, a weighted
one-mode network can be derived via projection, where the edge weights
indicate the number of times two nodes were observed in the same group. There
are many ways to handle these edge weights when the goal is to infer an
unweighted one-mode network (Borgatti and Halgin, 2011). In this experiment, I
compare the accuracy of inferences drawn using a simple approach to those
drawn using a state-of-the-art statistical backbone extraction model.

The simplest and most widely-used approach for handling edge weights in a
projection is to ignore them, and to focus on a simple unweighted projection.
In an unweighted projection, two nodes are connected if they were observed in
one or more of the same groups. This approach offers simplicity and
computational efficiency, but sets a low threshold for inferring that two
nodes are connected in the unobserved network of interest. Other, higher
thresholds can be used (e.g., observed in two, three, or more of the same
groups), but the choice of a given threshold is arbitrary. This approach also
ensures that the inferred network will be dense, with high levels of
transitivity and clustering, regardless of the true structure of the
unobserved network (Latapy et al., 2008; Neal, 2014; Watts, 2008), which may
diminish its accuracy.

Although methods have been proposed for choosing an edge weight threshold or
normalizing edges weights in a projection (Borgatti and Halgin, 2011), the
current state-of-the-art for obtaining an unweighted projection are
statistical backbone extraction models. These models use information from the
two-mode data (here, the observed groups) to define a statistical null model,
then test the statistical significance of each edge’s weight to determine
which should be retained in an unweighted ‘backbone.’ Many backbone extraction
models exist, however only two have preliminary evidence that they can
accurately infer unobserved networks: the stochastic degree sequence model
(SDSM) and fixed degree sequence model (FDSM; Neal et al., 2021, 2022;
Ferreira et al., 2022). Although both SDSM and FDSM are candidates for
inferring an unobserved network from observed groups, I consider only the
former because prior work has demonstrates they yield similar results (Neal et
al., 2021) and because FDSM is too computationally intensive to be useful in
practice (Godard and Neal, 2022).

The formal specification of the SDSM is described by Neal et al. (2021), but
like all statistical backbone extraction models it aims to determine when an
edge weight in a projection is statistically significantly larger than the
weight that would be expected in the projection of a random two-mode network.
The SDSM is distinguished from other backbone models by the information from
the two-mode network that it uses to evaluate the significance of an edge
weight. Specifically, it evaluates whether a given edge’s weight in a
projection is larger than expected in a random null model that simultaneously
controls for the degree sequences of both types of nodes. In this context, it
evaluates whether the number of group memberships shared by two individuals
(i.e. the edge weight in a projection) is larger than expected in a null model
that simultaneously controls for (a) the number of groups to which each of
those individuals belong and (b) the number of individuals that belong to each
group. By considering this information, the SDSM applies a unique threshold to
each edge. For example, observing two people in many of the same small groups
such as dinner parties provides stronger evidence for inferring they are
connected than observing them in many of the same large groups such as
concerts.

###  3.4 Experimental design and analysis

Table 1 summarizes the factorial experimental design, which varies 10
unobserved network structures, 6 numbers of observed groups, 6 probabilities
that groups correspond to cliques, and 2 inference methods, for a total of 720
experimental conditions. Within each condition, I compute the accuracy of the
inferred network as the similarity between the unobserved network and the
inferred network, averaged over 1000 replications. There are several ways to
measure the similarity of two networks. In the results below I report the
Pearson correlation coefficient, which is also known as the Matthews
correlation coefficient in the context of evaluating binary classifications
(i.e., is the edge present or absent?), because it is more robust than
alternate metrics (Chicco and Jurman, 2020). However, sensitivity analyses
confirm that other metrics, including Cohen’s $\kappa$ and the Jaccard
coefficient, yield the same results. The supplementary materials and the code
necessary to replicate all results reported below is available at
https://osf.io/6vcxa.

Factor | Levels
---|---
(A) Unobserved network | Random, Small World, Scale Free, Caveman, Core-Periphery
| Dolphin, Florentine, Karate, Law, Tailor
(B) Number of observed groups |  $N$, $2N$, $5N$, $10N$, $20N$, or $50N$
(C) Groups are cliques? | 50%, 60%, 70%, 80%, 90%, or 100%
(D) Method | Unweighted projection
| Stochastic degree sequence model (SDSM)
Replications per condition | 1000
Outcome | Mean Correlation, Jaccard, Kappa
Table 1: Summary of factorial experimental design

##  4 Results

Figure 2 reports the accuracy of a network inferred from observed groups using
an unweighted projection, by the structure of the unobserved network being
inferred (panels), number of groups observed (y-axis within panels), and
extent to which the observed groups correspond to cliques (x-axis within
panels). Similarly, Figure 3 reports the accuracy of a network inferred from
observed groups using a backbone extracted with the stochastic degree sequence
model. In both cases, the accuracy of the inferred network is measured using
the mean correlation between the unobserved ‘true’ network and the inferred
network over 1000 replications for the given experimental condition. Lighter
shades represent higher correlations, and thus conditions under which
inferences are more accurate.

Figure 2: Accuracy of a network inferred from observed groups using an
unweighted projection, by (a) the structure of the unobserved network being
inferred, (b) number of groups observed, and (c) extent to which the observed
groups correspond to cliques. Accuracy is measured using the correlation
between the unobserved and inferred networks. Figure 3: Accuracy of a network
inferred from observed groups using a backbone extracted with the stochastic
degree sequence model, by (a) the structure of the unobserved network being
inferred, (b) number of groups observed, and (c) extent to which the observed
groups correspond to cliques. Accuracy is measured using the correlation
between the unobserved and inferred networks.

Table 2 summarizes the experimental outcomes illustrated in Figures 2 and 3
via regression by predicting the accuracy of an inference as a function of the
unobserved network’s topology (size, density, transitivity, and number of
cliques) and characteristics of the observed groups (mean number of group
members per group, mean number of group memberships per person, number of
observed groups, and extent to which observed groups are cliques).
Unstandardized ($B$) and standardized ($\beta$) estimates are reported.
Standard errors and $p$-values are not reported because, in a simulation
context where sample size is arbitrary, they are not meaningful. When
interpreting these results below, I focus on the standardized estimates
because they indicate which properties have relatively more or less impact on
the accuracy of inferences.

| Unweighted Projection | SDSM Backbone
---|---|---
| $B$ | $\beta$ | $B$ | $\beta$
Intercept | -0.143 | — | 0.117 | —
Size | -0.001 | -0.034 | 0.004 | 0.224
Density | -0.150 | -0.041 | 0.234 | 0.060
Transitivity | -0.140 | -0.087 | 0.789 | 0.452
Number of Cliques |  $<$ 0.001 | 0.191 |  $<$ 0.001 | 0.211
Mean Group Members | -0.017 | -0.064 | -0.177 | -0.611
Mean Group Memberships | -0.001 | -0.256 | -0.003 | -0.499
Groups observed | -0.004 | -0.287 | 0.016 | 0.922
Groups are cliques? | 1.17 | 0.76 | 0.511 | 0.304
$R^{2}$ | 0.749 | 0.677
Table 2: Regression predicting the accuracy of an network inferred using a
given approach, as a function of the unobserved network’s topology and
characteristics of the observed groups.

Turning first to inferences drawn using a simple unweighted projection,
several patterns appear in Figure 2. First, as described by Guillaume and
Latapy (2004), when a large number of groups are observed, and those groups
directly correspond to cliques in the unobserved network, the unobserved
network can be inferred with perfect accuracy ($r=1$). Second, as expected,
inference accuracy declines when the observed groups are less likely to
correspond to cliques in the unobserved network. Third, unexpectedly,
inferences are not more accurate when they are based on a larger number of
observed groups. Instead, inferences are most accurate when they are based on
$2$-$5N$ observed groups. Inferences based on fewer observed groups are less
accurate because they draw on less information, while inferences based on more
observed groups are less accurate because they are overwhelmed by noisy
information. Finally, these patterns are consistent across all ten types of
unobserved network, suggesting that the structure of the unobserved network
plays little role in the accuracy of inferences drawn using an unweighted
projection.

These patterns are confirmed in the left panel of Table 2. The most important
factor in accurately inferring an unobserved network from observed groups
using an unweighted projection is the extent to which the observed groups
correspond to cliques ($\beta=0.76$). The more closely the observed groups
correspond to cliques, the more accurately a network can be inferred using an
unweighted projection. All other characteristics of the unobserved network and
observed groups have a limited impact on the accuracy of inferences.

Turning next to inferences drawn using a backbone extracted with the
stochastic degree sequence model, several patterns also appear in Figure 3.
First, as expected, the accuracy of inferences is higher when more groups are
observed. This occurs because, as with any statistical inference model,
inferences are more accurate when they are based on more data (here, when they
are based on more observed groups). Second, also as expected, the accuracy of
inferences is higher when the observed groups more closely correspond to
cliques in the unobserved network. Finally, there is some variation in the
accuracy of inferences for different unobserved networks. For example, across
all experimental conditions, an unobserved caveman network can be inferred
with high accuracy (mean $r=0.84$), while the Law Firm network can be inferred
with much lower accuracy (mean $r=0.38$).

These patterns are confirmed in the right panel of Table 2. The most important
factor in accurately inferring an unobserved network from observed groups
using an SDSM backbone is the number of groups observed ($\beta=0.922$). The
more groups that are observed, the more accurately a network can be inferred
using an SDSM backbone. Other factors play a more limited role. For example,
unobserved networks with higher transitivity can be inferred more accurately
($\beta=0.452$), which helps explain the generally higher accuracy with which
a caveman network can be inferred. Similarly, inferences are more accurate
when the setting is characterized by smaller groups ($\beta=-0.611$) and
individuals with fewer group memberships ($\beta=-0.499$).

Comparing the accuracy of inferences drawn using these two approaches suggests
that an SDSM backbone yields slightly more accurate inferences (mean $r=0.61$)
than an unweighted projection (mean $r=0.51$). However, there are significant
variations that impact when each approach is likely to yield an accurate
inference. The estimates in Table 2 indicate that when inferences are based in
more observed groups, the accuracy of an unweighted projection is reduced,
while the accuracy of an SDSM backbone is increased. Likewise, although the
extent to which groups correspond to clique has a large impact on the accuracy
of an unweighted projection, it plays a less significant role in the accuracy
of an SDSM backbone. The case of inferring an unobserved 50-node random
network serves to illustrate these differences. If 250 groups are observed
(i.e., $5N$) and those groups directly correspond to cliques in the unobserved
network (i.e., $p=1$), then an unweighted projection offers a very accurate
representation of the unobserved network ($r=0.97$), while an SDSM backbone is
less accurate ($r=0.73$). In contrast, if 2500 groups are observed (i.e.,
$50N$) but those groups do not correspond to cliques in the unobserved network
($p=0.5$), then an SDSM backbone offers a very accuracy representation of the
unobserved network ($r=0.93$), while an unweighted projection is much less
accurate ($r=0.18$).

##  5 Discussion

Practical and methodological challenges associated with collecting network
data directly from network members have led network researchers to develop
indirect data collection methods. Among the most widely used methods involves
attempting to infer an unobserved network from observed groups, for example,
inferring an unobserved network of collaboration from observed participation
on published papers. Although this approach is widely used, little is known
about when networks can be accurately inferred from observed groups.

In this paper, I conducted a series of experiments to examine how the accuracy
of a network inferred from observed groups depends on four factors: the
structure of the unobserved network to be inferred, the number of groups
observed, the extent to which the observed groups correspond to cliques in the
unobserved network, and the method used to draw inferences. The results
demonstrate that on average an unobserved network can be inferred from group
observations with moderate accuracy (mean $r=0.55$), but that there is
substantial variation in the expected accuracy of inferences under different
circumstances (range $r=1-0.05$).

These findings provide researchers with guidance about when an unobserved
network can be accurately inferred from observed groups. First, researchers
can use a simple unweighted projection to accurately (mean $r=0.84$) infer an
unobserved network of $N$ nodes if $2N$ to $5N$ groups are observed and
membership in those groups are believed to closely correspond to cliques in
the unobserved network of interest ($p\geq 0.9$). However, the ability to
accurately infer an unobserved network under these circumstances may not be
especially useful in practice because often observed groups will not perfectly
correspond to cliques in an unobserved network, and even if they did, it would
be impossible to know.

Second, researchers can use an SDSM backbone to accurately (mean $r=0.8$)
infer an unobserved network of $N$ nodes if at least $10N$ groups are
observed. Networks inferred using an SDSM backbone remain reasonably accurate
even when the membership of observed groups do not correspond to cliques in
the unobserved network (when $p\leq 0.6$, mean $r=0.7$). The ability to
accurately infer an unobserved network under these circumstances is useful in
practice because archival data sources mean the number of observed groups is
often much larger than the number of nodes (Borgatti and Halgin, 2011), and
because the relationship of groups to cliques is usually unknown.

Finally, researchers should not infer an unobserved network of $N$ nodes based
on $N$ or fewer observed groups because such inferences will be inaccurate
(mean $r=0.4$). This imposes an important scope condition on inferring
networks from observed groups, and limits the applicability of this approach
in some contexts. For example, despite serving as an early example (Breiger,
1974), it may be difficult to accurately infer a social network among 18 women
from just 14 social events. Similar issues arise in more contemporary multi-
level networks, where the number of scientists (nodes) exceeds the number of
disciplines (groups; Bellotti et al., 2016), the number of managers (nodes)
exceeds the number of organizations (groups; Brennecke and Rank, 2016), or the
number of students (nodes) exceeds the number of extra-curricular activities
(groups; Schaefer et al., 2022).

Although these results suggest that an unobserved network can be inferred from
observed groups under certain circumstances, this indirect approach to
measurement should be used with caution. When a network is measured directly
by asking network members a name generate question (e.g., who are your
friends) or from archival data (e.g., who do you follow online), the meaning
of edges in the network are explicitly known (friendship or following). In
contrast, when a network is inferred from observed groups, the meaning of
edges in the inferred network is ambiguous and depends on why (and, indeed,
whether) group co-membership suggests a relationship between two nodes.
Consider a network inferred from observations of groups of legislators
sponsoring bills, as is common in political network research. The edges in
such a network might be interpreted as indicating relationships of strategic
political alliances because co-sponsorship requires coordinated legislative
action, or of communication because co-sponsorship requires talking to one
another about bills, or merely of ideological similarity because co-
sponsorship indicates that two legislators favor the same bills (Neal, 2022).
Therefore, when a network is inferred from observed groups, the researcher
must offer a theory or rationale that observations of shared group memberships
provides a valid indicator of a particular type of relationship. Relatedly,
the researcher must also offer a theory or rationale that shared group
memberships observed over a given time period provides a valid indicator of a
given cross-sectional network.

As a first exploration into when an unobserved network can be inferred from
observed groups using projection-based methods, this study points to several
directions for future research. First, while inference using an SDSM backbone
seems promising, this method relies on frequentist $p$-values generated with
reference to a null model conditioned on two only characteristics of observed
groups: groups’ sizes and individuals’ number of memberships. Future research
may explore developing new backbone models that may improve inferential
accuracy by computing Bayesian likelihoods of edges’ existence, or by using an
ERGM framework to condition the null model on additional characteristics.
Second, these results are based on simulated observations of independent
groups. Future research may explore the accuracy of inferences from groups
that have been empirically observed, and from groups whose membership is not
independent.

A half-century ago, Breiger (1974) illustrated how a one-mode network could be
constructed from information about observed groups organized as a two-mode
network. This approach has since become widely used as a way to indirectly
measure one-mode networks that would be impractical or impossible to measure
directly. However, as an indirect measurement, it has been unclear whether
networks inferred from observed groups in this way are accurate, that is,
whether they correctly capture the structure of the unobserved network of
interest. These experimental results indicate that they can, thereby
vindicating the approach described by Breiger (1974) as a way to indirectly
measure networks. However, they also demonstrate that the degree of accuracy
depends on several factors, and therefore they also provide much-needed
guidance on when such an approach is appropriate.

## Data availability statement

The supplementary materials and the code necessary to replicate all results
reported below is available at https://osf.io/6vcxa.

## Funding statement

This work was supported by National Science Foundation awards #2016320 and
#2211744.

## Competing interests statement

None.

## References

  * adams (2020) j. adams.  _Gathering social network data_.  Sage, 2020.  https://doi.org/10.4135/9781544321486.
  * Andris et al. (2015) C. Andris, D. Lee, M. J. Hamilton, M. Martino, C. E. Gunning, and J. A. Selden.  The rise of partisanship and super-cooperators in the us house of representatives.  _PLOS One_ , 10(4):e0123507, 2015.  https://doi.org/10.1371/journal.pone.0123507.
  * Barabási and Albert (1999) A.-L. Barabási and R. Albert.  Emergence of scaling in random networks.  _Science_ , 286(5439):509–512, 1999.
  * Bellotti et al. (2016) E. Bellotti, L. Guadalupi, and G. Conaldi.  Comparing fields of sciences: multilevel networks of research collaborations in italian academia.  _Multilevel network analysis for the social sciences: Theory, methods and applications_ , pages 213–244, 2016.
  * Borgatti and Everett (2000) S. P. Borgatti and M. G. Everett.  Models of core/periphery structures.  _Social Networks_ , 21(4):375–395, 2000.  https://doi.org/10.1016/S0378-8733(99)00019-2.
  * Borgatti and Halgin (2011) S. P. Borgatti and D. S. Halgin.  Analyzing affiliation networks.  _The Sage Handbook of Social Network Analysis_ , 1:417–433, 2011.
  * Breiger (1974) R. L. Breiger.  The duality of persons and groups.  _Social Forces_ , 53(2):181–190, 1974.  https://doi.org/10.1093/sf/53.2.181.
  * Brennecke and Rank (2016) J. Brennecke and O. N. Rank.  Knowledge networks in high-tech clusters: A multilevel perspective on interpersonal and inter-organizational collaboration.  _Multilevel network analysis for the social sciences: Theory, methods and applications_ , pages 273–293, 2016.
  * Chicco and Jurman (2020) D. Chicco and G. Jurman.  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  _BMC Genomics_ , 21:1–13, 2020.  https://doi.org/10.1186/s12864-019-6413-7.
  * Erdős and Rényi (1959) P. Erdős and A. Rényi.  On random graphs.  _Publicationes Mathematicae_ , 6:290–297, 1959.
  * Feld (1981) S. L. Feld.  The focused organization of social ties.  _American Journal of Sociology_ , 86(5):1015–1035, 1981.  https://doi.org/10.1086/227352.
  * Ferreira et al. (2022) G. Ferreira, C. Henrique, F. Murai, A. P. Silva, M. Trevisan, L. Vassio, I. Drago, M. Mellia, and J. M. Almeida.  On network backbone extraction for modeling online collective behavior.  _PLOS One_ , 17(9):e0274218, 2022.  https://doi.org/10.1371/journal.pone.0274218.
  * Godard and Neal (2022) K. Godard and Z. P. Neal.  fastball: A fast algorithm to sample binary matrices with fixed marginals.  _Journal of Complex Networks_ , 10(6):cnac049, 2022.  https://doi.org/10.1093/comnet/cnac047.
  * Guillaume and Latapy (2004) J.-L. Guillaume and M. Latapy.  Bipartite structure of all complex networks.  _Information Processing Letters_ , 90(5):215–221, 2004.  https://doi.org/10.1016/j.ipl.2004.03.007.
  * Guimera et al. (2005) R. Guimera, B. Uzzi, J. Spiro, and L. A. N. Amaral.  Team assembly mechanisms determine collaboration network structure and team performance.  _Science_ , 308(5722):697–702, 2005.  https://doi.org/10.1126/science.1106340.
  * Kapferer (1972) B. Kapferer.  _Strategy and transaction in an African factory: African workers and Indian management in a Zambian town_.  Manchester University Press, 1972.
  * Karp (1972) R. M. Karp.  Reducibility among combinatorial problems.  In R. E. Miller, J. W. Thatcher, and J. D. Bohlinger, editors, _Complexity of Computer Computations_ , pages 85–103. Springer US, Boston, MA, 1972.  ISBN 978-1-4684-2001-2.  https://doi.org/10.1007/978-1-4684-2001-2_9.  URL https://doi.org/10.1007/978-1-4684-2001-2_9.
  * Kossinets (2006) G. Kossinets.  Effects of missing data in social networks.  _Social Networks_ , 28(3):247–268, 2006.  https://doi.org/10.1016/j.socnet.2005.07.002.
  * Krause et al. (2009) J. Krause, D. Lusseau, and R. James.  Animal social networks: an introduction.  _Behavioral Ecology and Sociobiology_ , 63:967–973, 2009\.  https://doi.org/10.1007/s00265-009-0747-0.
  * Latapy et al. (2008) M. Latapy, C. Magnien, and N. Del Vecchio.  Basic notions for the analysis of large two-mode networks.  _Social Networks_ , 30(1):31–48, 2008.  https://doi.org/10.1016/j.socnet.2007.04.006.
  * Lazega (2001) E. Lazega.  _The collegial phenomenon: The social mechanisms of cooperation among peers in a corporate law partnership_.  Oxford University Press, USA, 2001.
  * Lusseau et al. (2003) D. Lusseau, K. Schneider, O. J. Boisseau, P. Haase, E. Slooten, and S. M. Dawson.  The bottlenose dolphin community of doubtful sound features a large proportion of long-lasting associations: can geographic isolation explain this unique trait?  _Behavioral Ecology and Sociobiology_ , 54:396–405, 2003\.  https://doi.org/10.1007/s00265-003-0651-y.
  * Marsden (2011) P. V. Marsden.  Survey methods for network data.  In J. Scott and P. J. Carrington, editors, _The Sage Handbook of Social Network Analysis_ , pages 370–388. 2011.
  * Mizruchi (1996) M. S. Mizruchi.  What do interlocks do? an analysis, critique, and assessment of research on interlocking directorates.  _Annual Review of Sociology_ , 22(1):271–298, 1996.  https://doi.org/10.1146/annurev.soc.22.1.271.
  * Neal (2020a) J. W. Neal.  A systematic review of social network methods in high impact developmental psychology journals.  _Social Development_ , 29(4):923–944, 2020a.  https://doi.org/10.1111/sode.12442.
  * Neal et al. (2022) J. W. Neal, Z. P. Neal, and C. E. Durbin.  Inferring signed networks from preschoolers’ observed parallel and social play.  _Social Networks_ , 77, 2022.  https://doi.org/10.1016/j.socnet.2022.07.002.
  * Neal (2012) Z. Neal.  Structural determinism in the interlocking world city network.  _Geographical Analysis_ , 44(2):162–170, 2012\.  https://doi.org/10.1111/j.1538-4632.2012.00843.x.
  * Neal (2014) Z. P. Neal.  The backbone of bipartite projections: Inferring relationships from co-authorship, co-sponsorship, co-attendance and other co-behaviors.  _Social Networks_ , 39:84–97, 2014.  https://doi.org/10.1016/j.socnet.2014.06.001.
  * Neal (2020b) Z. P. Neal.  A sign of the times? weak and strong polarization in the us congress, 1973–2016.  _Social Networks_ , 60:103–112, 2020b.  https://doi.org/10.1016/j.socnet.2018.07.007.
  * Neal (2022) Z. P. Neal.  Constructing legislative networks in r using incidentally and backbone.  _Connections_ , 42:1–9, 2022.  https://doi.org/10.2478/connections-2019.026.
  * Neal (2023) Z. P. Neal.  The duality of networks and groups: Models to generate two-mode networks from one-mode networks.  _Network Science_ , pages 1–14, 2023.  https://doi.org/10.1017/nws.2023.3.
  * Neal et al. (2021) Z. P. Neal, R. Domagalski, and B. Sagan.  Comparing alternatives to the fixed degree sequence model for extracting the backbone of bipartite projections.  _Scientific Reports_ , 11:23929, 2021.  https://doi.org/10.1038/s41598-021-03238-3.
  * Newman (2004) M. E. Newman.  Coauthorship networks and patterns of scientific collaboration.  _Proceedings of the National Academy of Sciences_ , 101(suppl 1):5200–5205, 2004.  https://doi.org/10.1073/pnas.0307545100.
  * Padgett and Ansell (1993) J. F. Padgett and C. K. Ansell.  Robust action and the rise of the medici, 1400-1434.  _American Journal of Sociology_ , 98(6):1259–1319, 1993.  https://doi.org/10.1086/230190.
  * Peel et al. (2022) L. Peel, T. P. Peixoto, and M. De Domenico.  Statistical inference links data and theory in network science.  _Nature Communications_ , 13(1):6794, 2022.  https://doi.org/10.1038/s41467-022-34267-9.
  * Schaefer et al. (2010) D. R. Schaefer, J. M. Light, R. A. Fabes, L. D. Hanish, and C. L. Martin.  Fundamental principles of network formation among preschool children.  _Social Networks_ , 32(1):61–71, 2010.  https://doi.org/10.1016/j.socnet.2009.04.003.
  * Schaefer et al. (2022) D. R. Schaefer, T. V. Khuu, J. A. Rambaran, D. Rivas-Drake, and A. J. Umaña-Taylor.  How do youth choose activities? assessing the relative importance of the micro-selection mechanisms behind adolescent extracurricular activity participation.  _Social Networks_ , 2022.  https://doi.org/10.1016/j.socnet.2021.12.008.
  * Wang et al. (2012) D. J. Wang, X. Shi, D. A. McFarland, and J. Leskovec.  Measurement error in network data: A re-classification.  _Social Networks_ , 34(4):396–409, 2012.  https://doi.org/10.1016/j.socnet.2012.01.003.
  * Watts (1999) D. J. Watts.  Networks, dynamics, and the small-world phenomenon.  _American Journal of Sociology_ , 105(2):493–527, 1999.  https://doi.org/10.1086/210318.
  * Watts (2008) D. J. Watts.  _Six Degrees: The Science of a Connected Age_.  W. W. Norton, 2008.
  * Watts and Strogatz (1998) D. J. Watts and S. H. Strogatz.  Collective dynamics of ‘small-world’networks.  _Nature_ , 393(6684):440–442, 1998.  https://doi.org/10.1038/30918.
  * Wellman (1988) B. Wellman.  Structural analysis: From method and metaphor to theory and substance.  In B. Wellman and S. D. Berkowitz, editors, _Social structures: A network approach_ , pages 19–61. Cambridge University Press, 1988.
  * Zachary (1977) W. W. Zachary.  An information flow model for conflict and fission in small groups.  _Journal of Anthropological Research_ , 33(4):452–473, 1977.  https://doi.org/10.1086/jar.33.4.3629752.