# Towards Perceived Security, Perceived Privacy, and the Universal Design of
E-Payment Applications

Urvashi Kishnani, Isabella Cardenas, Jailene Castillo11footnotemark: 1,
Rosalyn Conry11footnotemark: 1,
Lukas Rodwin11footnotemark: 1, Rika Ruiz11footnotemark: 1, Matthew
Walther11footnotemark: 1, and Sanchari Das
University of Denver These authors contributed equally to this project

### Abstract

With the growth of digital monetary transactions and cashless payments,
encouraged by the COVID-19 pandemic, use of e-payment applications is on the
rise. It is thus imperative to understand and evaluate the current posture of
e-payment applications from three major user-facing angles: security, privacy,
and usability. To this, we created a high-fidelity prototype of an e-payment
application that encompassed features that we wanted to test with users. We
then conducted a pilot study where we recruited $12$ participants who tested
our prototype. We find that both security and privacy are important for users
of e-payment applications. Additionally, some participants perceive the
strength of security and privacy based on the usability of the application. We
provide recommendations such as universal design of e-payment applications.

##  1 Introduction

E-payment applications (such as Google Pay, Apple Pay, Cash App, PayPal, or
Venmo) allow for contactless transactions, giving people opportunities to buy
goods or services, or transfer money when traditional payment methods are
unsafe or unavailable. According to a $2019$ report by Mckinsey and Company,
about $77\%$ of people made a mobile payment in the year $2019$ [2]. As a
major financial system in modern economy, e-payment systems must aim to be
absolutely secure with financial information and accounts. Users of e-payment
systems not only face certain financial risks, but are also prone to privacy
risks. This is because most e-payment applications require critical
information such as Personally Identifiable Information (PII) of the users,
including name, date of birth, address, and banking information. Thus, users
are susceptible to potential loss of PII, introducing the threat of identity
theft [5]. Additionally, usability and design of e-payment applications is
also equally important, as prior studies have shown that usability of mobile
application plays a role in the perceived security and privacy understanding
by the user [16].

##  2 Methodology

###  2.1 Prototype Building

We make one comprehensive prototype to test various functionalities and
behaviors that were previously studied separately (such as [3, 7, 5, 9, 10]).
Unlike most payment apps, our prototype presents the option to view the
privacy policy prominently. Moreover, instead of a lengthy policy that users
usually do not read, we present a short simplified privacy policy screen to
gauge the engagement of users. Lastly, unlike most other payment apps, we
provide mandatory Multi-Factor Authentication (MFA) options to our users at
the time of login and registration. Additionally, our prototype incorporates
four different MFA options and gives users the choice to select their
preferred method, thereby improving security while still allowing users to
feel in control. The prototype is built using Figma, which is free for
students and educators 111https://www.figma.com/education/. The prototype has
several interlinked pages implementing security, privacy, and usability
concepts. The prototype consists of a total of $17$ screens. The prototype is
built to only be intractable with buttons and does not allow for any text-
based input. This ensures that we do not accidentally collect any personal
data from the participants during our study.

(a)

(b)

(c)

(d)

Figure 1: Screenshots of Prototype (Left-to-Right: Registration Page,
Permissions Page, Home Screen, Settings Page.)

###  2.2 User Study

Recruitment and Eligibility: Recruitment information including an eligibility
criteria and link to a pre-screening survey was distributed using flyers,
emails, social media posts. To be eligible to participate in the study,
participants had to meet the following requirements: Participant must (1) be
able to come to the university campus to perform the in-lab study, (2) have
experience in using at least one e-payment or banking application, and (3) be
$18$ years of age or older and must reside in the US. Those participants that
consented to the study and met all the requirements proceeded to fill in our
pre-screening survey based on survey instrument proposed by Rajivan et al.
[11]. From the $16$ respondents, we selected $N=12$ participants who were then
invited to an in-person study where they first tested and interacted with a
prototype and then answered questions to a semi-structured interview. The
study with each participant took $20-30$ minutes.

Interview and Qualitative Analysis: The study was conducted in the university
campus by the primary researcher of this work. At the time of the interview,
the participant were given an overall briefing about the task, and then tested
and interacted with the application prototype. After this, they were asked to
answer questions to a semi-structured interview about the usability, privacy,
and security features of the application. There was no compensation for
participation. To facilitate the qualitative analysis of the interviews, each
interview was audio recorded if appropriate consent was given by the
participant. These interviews were then transcribed using Zoom and Google
Pixel $7$ Pro’s automatic live transcription services to convert the audio to
textual context. Two researchers open coded two interviews to generate
keywords and phrases of interest and grouped these codes on two-levels to form
the codebook. Thereafter, we conducted a thematic analysis of the remaining
transcribed interviews.

##  3 Results

Table 1 summarizes the demographics of our participants. Our participants,
being university students, represented a relatively younger population. The
distribution of education level is in agreement with the participants’ ages.
We discuss themes and corresponding sub-themes from our thematic analysis
below.

Table 1: Participant demographics (NB: Non-Binary, NS: Not Specified) ID | Age | Gender | Education Level | Ethnicity
---|---|---|---|---
P01 | 18-24 | Male | High School | White
P02 | 18-24 | Male | High School | White
P03 | 18-24 | Male | Diploma | White
P04 | 18-24 | Male | Bachelors | White
P05 | 18-24 | Male | Diploma | White
P06 | 18-24 | Female | High School | White
P07 | 18-24 | Female | Diploma | White
P08 | 25-30 | Male | Bachelors | NS
P09 | 18-24 | NB | Bachelors | White
P10 | 18-24 | Female | Bachelors | White
P11 | 18-24 | Female | Bachelors | Asian
P12 | 18-24 | Male | High School | Hispanic

###  3.1 Security

Perceived Security:  Seven participants admitted that they find it difficult
to comment on the security of the application and are unsure whether it is
secure or not. Two participants mentioned that the blue colored theme
represented security to them. Participant P02 mentioned that the more
complicated an application, the more problems that can create. Participants
also mentioned how the usability of the application impacted their view on the
security.

Multi-factor Authentication:  Our prototype and interview questions focused on
two applications of Multi-Factor Authentication (MFA). The first is during
registration/login, and the second is before sending money. Eight participants
felt that the availability of various MFA options during login meant that the
application was more secure. Three participants mentioned that MFA security
was more important than the convenience of a faster login. Five participants
mentioned that they preferred OTP options, and one explicitly mentioned that
they liked OTP over biometrics. Three participants said that they found the
use of FaceID to be concerning for payment applications and two expressed that
they had faced some usability barrier in using FaceID. On the other hand,
three participants said they preferred biometric options over OTP, three
participants said that they find FaceID convenient, and one participant
explicitly said that they did not like OTP. Moreover, three participants said
they find app-based MFA options (like Duo) annoying.

Login Notifications:  Most participants responded positively to login
notifications. Nine participants said they really liked them and found them to
be important, two participants found them annoying and one participant said
that they never received them. Regarding actions that the participants take
when they receive login notifications, seven participants also mentioned that
they don’t take any action if they know the notification was triggered because
of them or if they see the notification from the area that they are in. Seven
participants also mentioned that if they recognize that the login was not made
by them, they take action to update their information or change their
passwords. One participant offered additional recommendation to enhance the
functionality of login notifications.

Adding Beneficiaries using QR Codes:  Six participants appreciated the
additional layer of identity verification that adding beneficiaries using QR
codes provided. Five mentioned that they have used it a few times and two said
that they use it frequently. However, four participants said that this was a
new concept to them. But three mentioned that even though they haven’t use
this, they can understand how it can be beneficial. Participant in favor of QR
Codes mentioned this help increase usability of the application.

###  3.2 Privacy

Perceived Privacy:  Five participants said that they found the app was built
while keeping users privacy in mind. They mentioned different reasons for this
opinion. Some said that they felt the option to add beneficiaries by QR code
meant that the app was built with a privacy lenses. Some said that the prompts
provided by the application for the prompts for permissions requested for
camera of fingerprint indicated that the app respected user’s privacy. One
participant said that the app respected their privacy as they were not asked
to write messages while sending money as they found this option intrusive in
Venmo. Three participants said that they did not find this application was
built for user’s privacy and they mentioned various reasons for this. Some
said that they found that there were no options to toggle privacy settings.
Some participants mentioned that they required more transparency on the third
parties that were involved. One participants said that they do not trust
payment applications due to the financial interest of companies. Two
participants said that there was no reason to not trust the app. We note that
two participants found the app to be more private due to the authentication
features, although that is more geared towards security.

Application Permissions:  Six participants say that they scrutinize
permissions overall. Participants mainly focused on three major permissions:
location, contacts and camera. Two participants also mentioned that permission
pop-ups are annoying but they find that they are okay for crucial applications
like e-payment or banking. Four participants mentioned that the specifically
scrutinize location permissions. One participant mentioned that they feel
location is helpful to know when a transaction was made and can be used to
prevent scams if we transaction was made in an area where the user was not
present. Four participants mentioned that this scrutinized contacts. Two
participant mentioned that they find sharing contacts is inconvenient
especially because they have a large number of contacts and they don’t want
all of them to be imported into the payment application. Four, participants
mentioned that this scrutinized camera and that they would allow camera for
functional reasons if required by the application, like scanning QR code or
taking pictures of cheque. Finally when asked about the permissions requested
by the prototype, the participants felt that they were pretty standard and
something that they would expect.

Privacy Policies:  When asked about the participants experience with privacy
policies in general, $9$ out of $12$, participants said that they do not read
privacy policies. Only one participant mentioned that they do read privacy
policy and that they understand that too. Three participants also mentioned
that they do not understand privacy policies and three participants mentioned
that they do understand privacy policies. When asked about the privacy
policies of the prototype, eight participants mentioned that they just read,
skimmed, or glanced over the policy for the prototype whereas five mentioned
that they did not read the privacy policies for the prototype. When asked,
what if they were easily able to find the privacy policy, six participants
said that they were able to find that easily, whereas only one participant
said that they felt the privacy policy option was hard to find or quite
hidden.

Social Features:  Social features of payment applications, like Venmo which
provides a social media-like feed of transactions, were mostly received with a
negative sentiment. Six participants found this concept to be weird or strange
and five participants said that they prefer private transactions. Four
participants also mentioned they find social features to be intrusive and four
participants said that there’s no need to make payment apps like a social
media app. Three participants said that it’s strange to give access to
contacts in the first place when it comes to payment applications. Two
participants said that the default should be opt-out and the participants
should have the option to opt-in if required, instead of it being the other
way around. One participant mentioned that such social features can actually
act as identity verification. Finally, two participants said that they had no
opinions on social features for e-payment applications.

###  3.3 Usability

Overall User Experience:  When it came to the overall user experience of using
the prototype, nine participants said that they found it easy to use. Five
said that the application was quite simple and five others said that they
found it quite intuitive to use. One participant said that they neither liked
nor disliked the overall usability of the application. Eight participants said
that they like the usability factor of using QR codes as it make it much
easier for them to find someone by scanning another person’s QR code as
compared to finding someone by name or typing complicated usernames. They also
said that this would help avoid mistakes and would make sure that the money is
sent to the right person. One participant pointed out that there was no option
to withdraw money to their bank, and that this is one option that they use
regularly and they said that it was missing from this prototype.

Navigation:  When asked about the experience of navigating, the application
nine participants found that it was simple and easy to navigate. They like
having important features up front in the application, such as transactions on
the home-screen immediately after logging in. However, some participants also
felt that shortcuts like a pop-up menu under the profile picture, a navigation
bar at the bottom with a home button would help make navigation easier. One
participant mentioned that they were confused that after they registered, it
went straight to the home screen instead of asking them to login again after
registration.

Font:  Only three participants said that the font on the prototype was easy-
to-read. Two participant mentioned that they would change the font in a few
places, for example, by adding outline to text to make it more legible. When
asked about the option to change font size or type, most participants said
that they wouldn’t bother using such a feature. However, five participant did
say that they would appreciate such a feature and two participants
specifically mentioned that increasing the font size would be helpful to older
adults or people with low vision, which would make the app more inclusive.

Icons:  Four participants said they enjoyed icons for navigability, but one
warned that a lot of icons could make an app cluttered. With the consensus
being that some amount of icons for important actions are best. For the
prototype, four participants didn’t notice many icons and said that more icons
could be added. Two participants said that the icons could be made a little
smaller. Participants also mentioned that the icons used should be universal
and well known.

##  4 Discussions and Implications

Reduce Social Features: Most of our participants did not have positive
opinions on Social features for E-Payment Applications where other user’s can
view their transactions. One participant also finds the concept of adding a
note to each transaction to be intrusive. Users expressed disinterest in
viewing transactions of others. One user suggested that such a concept can
contribute to gamification of E-Payment applications where users may be pushed
to send or receive more money. When asked about whether our participants used
this option on an application like Venmo, all of our users said that they have
not set their transactions to be public. Some users said that they only share
their transactions with close family and friends. Social features like these
also require users to share permission to their contacts, which allows apps to
add “friends" for the users. However, our participants also expressed concern
with sharing all of their contacts with a payment application, and would
prefer to only share selected contacts on need basis. Prior work has shown
that social features of apps like Venmo can cause security issues [8]. Our
results of participants feeling indifferent or awkward about seeing other’s
transaction corroborates with prior research [1].

Fine-Grained Options for User’s Settings: Users like to be know that they are
in control of their privacy, this makes them have more trust in the
application that they are using. Most users expressed that multiple two-factor
authentication options made them feel more secure. Some users preferred
Biometric Authentication like Fingerprint and FaceID, over One Time Passwords
(OTP) like Email or SMS, and others had the opposite preference. So having all
these options presented to the user was received positively. Users also
expressed concerns for a missing page in the prototype where they could fine-
tune more settings. Further, one user said that giving permission to camera
separately for each feature (like FaceID and scanning QR codes) was preferred,
as compared to giving the permission one time for the entire app. Prior
studies agree with user perception to be in favor of two-factor authentication
as compared to single-factor authentication [6].

Universal Design for Payment Apps: Most participants agreed with the
implementation of a blue and white theme for the payment application. Users
mentioned that they like the use of a cool and soothing color like blue for a
payment application because it made them feel safer. Users also mentioned that
they did not like the choice of yellow text over blue elements and expressed
concerns for contrast and accessibility. Thus, universal design practices can
make the app more accessible and inclusive [4, 15, 12]. All users said that
they found our prototype easy to navigate and liked having all the important
information for a payment application presented upfront. Additionally, the use
of shortcuts like a navigation bar will help make the users’ navigation
experience more seamless. Users mentioned that using recognized icons can help
with the comprehension of the application elements. Similar to our results,
prior studies have shown that a minimalist application on-boarding process
increased user engagement [13, 14].

##  5 Future Work and Limitations

As a future extension to this study, the prototype can be enhanced in two
ways. First, by implementing the feedback from our participants to improve the
functionality and usability of the prototype. Second, by adding extra
functionality to the prototype and making it closer to a real world payment
application. These improvements can facilitate a more seamless experience
during user testing and overcome user bias in reporting on a prototype.
Moreover, the user study can be modified to ask users to perform certain tasks
on the applications. By using a think-aloud protocol and timing the users on
the tasks, we can understand the pain-points of the prototype. Further,
usability of the prototype can be measured using the System Usability Scale
(SUS). Such improvements can lead to richer findings and a more comprehensive
understanding of user perception of our prototype.

##  6 Conclusion

In this study, we built a prototype of e-payment application using Figma and
recruited $N=12$ participants for prototype testing followed by a semi-
structured interview. We find that the usability of a mobile application
informs important perception on the security and privacy for the users. Users
pay attention to ease-of-use and aesthetics of the applications, such as use
of soothing and cool colors, that make them feel safe. We provide
recommendations from our study and directions for potential future extensions.

## References

  * [1] Caraway, M., Epstein, D. A., and Munson, S. A. Friends don’t need receipts: The curious case of social awareness streams in the mobile payment app venmo.  Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 1–17.
  * [2] Company, M. . Are convenience and rewards leading to a digital flashpoint? - insights from mckinsey’s 2019 digital payments survey, Oct 2019.
  * [3] De, P., Dey, K., Mankar, V., and Mukherjea, S. An Assessment of QR Code as a User Interface Enabler for Mobile Payment Apps on Smartphones.  In Proceedings of the 7th Indian Conference on Human-Computer Interaction (New York, NY, USA, Dec. 2015), IndiaHCI ’15, Association for Computing Machinery, pp. 81–84.
  * [4] Fogli, D., Arenghi, A., and Gentilin, F. A universal design approach to wayfinding and navigation.  Multimedia Tools and Applications 79 (2020), 33577–33601.
  * [5] Ho, K. K. W., See-To, E. W. K., and Chiu, D. K. W. “Price Tag” of Risk of Using E-Payment Service.  Journal of Internet Commerce 19, 3 (July 2020), 324–345.  Publisher: Routledge _eprint: https://doi.org/10.1080/15332861.2020.1742482.
  * [6] Khaskheli, G. M., Sherbaz, M., and Shaikh, U. R. A comparative usability study of single-factor and two-factor authentication.  Tropical Scientific Journal 1, 1 (2022), 17–27.
  * [7] Kim, C., Tao, W., Shin, N., and Kim, K.-S. An empirical study of customers’ perceptions of security and trust in e-payment systems.  Electronic Commerce Research and Applications 9, 1 (Jan. 2010), 84–95.
  * [8] Kraft, B., Mannes, E., and Moldow, J. Security research of a social payment app, 2014.
  * [9] Ming-Yen Teoh, W., Choy Chong, S., Lin, B., and Wei Chua, J. Factors affecting consumers’ perception of electronic payment: an empirical analysis.  Internet Research 23, 4 (Jan. 2013), 465–485.
  * [10] Ogbanufe, O., and Kim, D. J. Comparing fingerprint-based biometrics authentication versus traditional authentication methods for e-payment.  Decision Support Systems 106 (Feb. 2018), 1–14.
  * [11] Rajivan, P., Moriano, P., Kelley, T., and Camp, L. J. Factors in an end user security expertise instrument.  Information & Computer Security 25, 2 (2017), 190–205.
  * [12] Reid, G., Strnadová, I., and Cumming, T. Expanding horizons for students with dyslexia in the 21st century: universal design and mobile technology.  Journal of Research in Special Educational Needs 13, 3 (2013), 175–181.
  * [13] Strahm, B., Gray, C. M., and Vorvoreanu, M. Generating mobile application onboarding insights through minimalist instruction.  In Proceedings of the 2018 Designing Interactive Systems Conference (2018), pp. 361–372.
  * [14] Surendran, S., Sivaselvan, B., and Oswald, C. Emergent user design framework for e payment mobile application.  In 2018 International Conference on Computer, Communication, and Signal Processing (ICCCSP) (2018), IEEE, pp. 1–6.
  * [15] Walker, B. N., Tomlinson, B. J., and Schuett, J. H. Universal design of mobile apps: Making weather information accessible.  In Universal Access in Human–Computer Interaction. Design and Development Approaches and Methods: 11th International Conference, UAHCI 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9–14, 2017, Proceedings, Part I 11 (2017), Springer, pp. 113–122.
  * [16] Özkan, S., Bindusara, G., and Hackney, R. Facilitating the adoption of e-payment systems: theoretical constructs and empirical analysis.  Journal of Enterprise Information Management 23, 3 (Jan. 2010), 305–325.

##  Appendix A Pre-Screening Survey

Initial Screening

  1. 1.

What is your age?

  2. 2.

What is your email address?

  3. 3.

Select the e-payment or mobile banking applications that you have used utilize
(choose at least one) $\circ$ PayPal $\circ$ Venmo $\circ$ CashApp $\circ$
Apple Pay $\circ$ Google Pay $\circ$ Samsung Pay $\circ$ Meta Pay $\circ$
Zelle $\circ$ Banking app (please specify) $\circ$ other e-payment app (please
specify)

Everyday Digital Usage

  1. 1.

"I often ask others for help with the computer or mobile phone." $\circ$
Strongly Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree
$\circ$ Somewhat Agree $\circ$ Strongly Agree

  2. 2.

"Others often ask me for help with the computer or mobile phone." $\circ$
Strongly Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree
$\circ$ Somewhat Agree $\circ$ Strongly Agree

  3. 3.

Which of the following things have you done? (multiple-select) $\circ$
Designed a website $\circ$ Designed a mobile app $\circ$ Registered a domain
name $\circ$ Used SSH $\circ$ Configured a firewall $\circ$ Created a database
$\circ$ Installed a computer program $\circ$ Written a computer program
$\circ$ Deployed an application on a mobile app store $\circ$ None of the
above

  4. 4.

Do you have an up-to-date virus scanner on your computer? $\circ$ Yes $\circ$
No

  5. 5.

Have you taken or taught a course on computer security? $\circ$ Yes $\circ$ No

  6. 6.

Is computer security one of your main job responsibilities? $\circ$ Yes
$\circ$ No

  7. 7.

Do you have a security cognate or do you plan on security being a feature of
your job role? $\circ$ Yes $\circ$ No

  8. 8.

To your knowledge, have you ever been hacked or compromised by attackers?
$\circ$ Yes $\circ$ No $\circ$ I don’t know

  9. 9.

Have you ever suffered data loss for any reason? (ex: Hacking, data
corruption, hard drive failure) $\circ$ Yes $\circ$ No $\circ$ I don’t know

  10. 10.

"I can control the risk of my online account being taken over or being
phased." $\circ$ Strongly Disagree $\circ$ Somewhat Disagree $\circ$ Neither
Agree nor Disagree $\circ$ Somewhat Agree $\circ$ Strongly Agree

  11. 11.

"The harm of being locked out of my account would be immediate." $\circ$
Strongly Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree
$\circ$ Somewhat Agree $\circ$ Strongly Agree

  12. 12.

"If I am exposed to online risk, I can mitigate the harm." $\circ$ Strongly
Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree $\circ$
Somewhat Agree $\circ$ Strongly Agree

  13. 13.

"Experts understand phishing and know how to protect me from it." $\circ$
Strongly Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree
$\circ$ Somewhat Agree $\circ$ Strongly Agree

  14. 14.

"I understand phishing and know how to protect myself." $\circ$ Strongly
Disagree $\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree $\circ$
Somewhat Agree $\circ$ Strongly Agree

  15. 15.

"Phishing is widespread and affects many people, and not only for target
individuals." $\circ$ Strongly Disagree $\circ$ Somewhat Disagree $\circ$
Neither Agree nor Disagree $\circ$ Somewhat Agree $\circ$ Strongly Agree

  16. 16.

"Phishing is everywhere. It is a constant exposure." $\circ$ Strongly Disagree
$\circ$ Somewhat Disagree $\circ$ Neither Agree nor Disagree $\circ$ Somewhat
Agree $\circ$ Strongly Agree

  17. 17.

"Phishing is a new kind of risk." $\circ$ Strongly Disagree $\circ$ Somewhat
Disagree $\circ$ Neither Agree nor Disagree $\circ$ Somewhat Agree $\circ$
Strongly Agree

Demographics

  1. 1.

What ethnicity do you identify with? (Please select all that apply) $\circ$
American Indian or Native American $\circ$ Asian $\circ$ Native Hawaiian or
Other Pacific Island $\circ$ White $\circ$ Hispanic $\circ$ Other. Please
Specify $\circ$ Do not Wish to Specify

  2. 2.

Which gender do you identify with the most? $\circ$ Female $\circ$ Male
$\circ$ Transgender $\circ$ Non-binary $\circ$ A gender not listed here
$\circ$ Do not Wish to Specify

  3. 3.

What is the highest level of education you have completed? (If currently
enrolled, highest degree received.) $\circ$ Less than high school $\circ$ High
school graduate $\circ$ Diploma $\circ$ Vocational training $\circ$ Bachelors
degree program $\circ$ Masters degree program $\circ$ Professional degree
$\circ$ Doctorate $\circ$ Other. Please Specify

##  Appendix B Interview Questions

Security

  1. 1.

Do you think the app is secure or does it have some vulnerabilities? What
security features did you notice? What potential vulnerabilities or issues did
you notice?

  2. 2.

What are your opinions on Multi-Factor Authentication (MFA) options like
biometric authentication, One Time Passwords (OTP), etc. while logging in?
Have you used face id for payment? What are your thoughts on that?

  3. 3.

What are your opinions on MFA options before sending money?

  4. 4.

What is your opinion of the feature that allows you to add beneficiaries using
QR codes? Have you heard of this before? Have you used it before?

  5. 5.

What is your opinion on receiving login notifications? Do you take any steps
when you receive them?

Privacy

  1. 1.

Do you think the app is designed while keeping user’s privacy in mind? Why or
why not?

  2. 2.

What did you think about the permissions requested by the application? Do you
usually scrutinize permissions for applications, or do you just accept
whatever is requested?

  3. 3.

Were you able to find Privacy Policies? Did you read them? Do you usually read
and understand privacy policies?

  4. 4.

What do you think about the identity verification procedures for payment
applications where you send a document like Driver’s License or SSN number to
verify you as a user? Do you think that is helpful or redundant?

  5. 5.

What is your opinion on social features of e-payment the applications such as
Venmo?

Usability

  1. 1.

What is your overall opinion on the experience of using the application? Is
there something you liked or disliked about it?

  2. 2.

What did you think about the theme and color scheme of the application? Is
there anything you’d like to add or change?

  3. 3.

What did you think about the font used in the application? Do you like to have
the ability to change font type or size?

  4. 4.

What did you think about the icons used in the application? Do you think the
icons help or do you think the application would be better without them?

  5. 5.

What did you think about the Graphical User Interface (GUI) organization and
layout of the application? Was it easy to use? Was it intuitive? Did it take
you a long time to familiarize yourself with the application?


#  RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT
Analysis.

Xiaoman Zhang  Shanghai Jiao Tong University, Shanghai, China  University of
Science and Technology of China, Anhui, China  Chaoyi Wu  Shanghai Jiao Tong
University, Shanghai, China  University of Science and Technology of China,
Anhui, China  Ziheng Zhao  Shanghai Jiao Tong University, Shanghai, China
University of Science and Technology of China, Anhui, China  Jiayu Lei
University of Science and Technology of China, Anhui, China
Ya Zhang  Shanghai Jiao Tong University, Shanghai, China  University of
Science and Technology of China, Anhui, China  Yanfeng Wang  Shanghai Jiao
Tong University, Shanghai, China  University of Science and Technology of
China, Anhui, China  Corresponding Authors  Weidi Xie  Shanghai Jiao Tong
University, Shanghai, China  University of Science and Technology of China,
Anhui, China  Corresponding Authors

###### Abstract

Developing generalist foundation model has recently attracted tremendous
attention among researchers in the field of AI for Medicine (AI4Medicine). A
pivotal insight in developing these models is their reliance on dataset
scaling, which emphasizes the requirements on developing open-source medical
image datasets that incorporate diverse supervision signals across various
imaging modalities. In this paper, we introduce RadGenome-Chest CT, a
comprehensive, large-scale, region-guided 3D chest CT interpretation dataset
based on CT-RATE. Specifically, we leverage the latest powerful universal
segmentation and large language models, to extend the original datasets (over
25,692 non-contrast 3D chest CT volume and reports from 20,000 patients) from
the following aspects: (i) organ-level segmentation masks covering 197
categories, which provide intermediate reasoning visual clues for
interpretation; (ii) 665 K multi-granularity grounded reports, where each
sentence of the report is linked to the corresponding anatomical region of CT
volume in the form of a segmentation mask; (iii) 1.3 M grounded VQA pairs,
where questions and answers are all linked with reference segmentation masks,
enabling models to associate visual evidence with textual explanations. All
grounded reports and VQA pairs in the validation set have gone through manual
verification to ensure dataset quality. We believe that RadGenome-Chest CT can
significantly advance the development of multimodal medical foundation models,
by training to generate texts based on given segmentation regions, which is
unattainable with previous relevant datasets. We will release all segmentation
masks, grounded reports, and VQA pairs to facilitate further research and
development in this field.

## Background & Summary

In the recent literature, the evolution of large-scale foundation models [1,
2, 3, 4, 5] has sparked significant interest in the development of generalist
medical AI (GMAI) systems [6, 7, 8, 9, 10], particularly within the realm of
radiology—a crucial component of medical diagnostics. By training on large-
scale visual-language medical datasets, i.e., medical scans paired with global
clinical reports, for example, MIMIC-CXR [11] has chest X-ray scans from
227,835 studies, and CT-RATE [12] contains chest CT scans from 20,000
patients. These medical models have demonstrated the preliminary ability for
writing clinical reports, aiming to support radiologists throughout their
workflow and markedly reducing workloads.

However, existing datasets only provide global reports for the medical scan,
which has posed limitations on training models that enables grounded report
generation, grounded question answering, i.e., describing regional
abnormalities and relevant normal findings, or answer questions corresponding
to certain regions. To further push forward the training of more capable
generalist models, we propose to extend the existing image-reports datasets
with region-wise description, i.e., linking the descriptive labels or findings
from diagnostic reports to their corresponding anatomical regions in the
images, in the form of segmentation masks for explainability.

In this paper, we introduce RadGenome-Chest CT, a comprehensive, large-scale
and fine-grained annotated dataset for 3D chest CT interpretation, built upon
the publicly available CT-RATE [12]. Initially, we employ the latest powerful
text-prompted universal segmentation model, SAT [13], to segment primary
anatomical targets in the image. Subsequently, utilizing large language models
and NER models, we break all reports into an anatomically hierarchical
structured format, and link the reports’ sentences to visual regions in CT
volume. Finally, we further generate visual question-answering pairs closely
related to the structured report and segmented image, from both region level
and case level. In summary, we have extended the original image-report
datasets from the following aspects:

  1. i)

Organ-level segmentation masks that covers 197 categories, i.e., all the
critical regions existing in clinical CT reports;

  2. ii)

665k multi-granularity grounded reports, with each sentence grounded to the
corresponding anatomical region.

  3. iii)

1.3 M grounded VQA pairs, concerning both critical region-wise findings and
comprehensive case-wise impressions. All the questions and answers are linked
to segmentation masks for reference.

We believe that RadGenome-Chest CT , with provided region-to-report
associations, can significantly advance the development of agent-based
multimodal medical foundation models, that enables to generate texts, grounded
on the corresponding visual regions, which is unattainable with previous
relevant datasets.

Figure 1: Data construction pipeline of RadGenome-Chest CT.

## Methods

In this section, we start by introducing the source dataset that RadGenome-
Chest CT is built on. Next, we provide a detailed description of the
collection procedure for obtaining segmentation masks, and region-wise
reports, as illustrated in Fig. 1. The outcomes from each step are presented
in Fig. 2.

### Data Source

We initiate our study with CT-RATE [12]
(https://huggingface.co/datasets/ibrahimhamamci/CT-RATE), it is a dataset of
25,692 non-contrast 3D chest CT volumes derived from 21,304 unique patients,
each volume is accompanied by a radiology text report and annotated with 18
distinct types of abnormalities. These 25,692 non-contrast 3D chest CT volumes
have been reconstructed with various methods to accommodate different window
settings, totaling 50,188 images. For consistency in this paper, we have
standardized all CT volumes to a uniform voxel spacing of $1\times 1\times
3mm$, resulting in only 25,692 paired CT volumes and reports. We follow the
official division: 20,000 patients ( 24,128 volumes) were allocated to
training and 1,304 (1,564 volumes) for validation.

### Constructing RadGenome-Chest CT

The pipeline consists of three major stages, as shown in Fig. 1: (i)
segmentation mask generation, where detailed masks for each anatomical region
in the chest CT volumes are created; (ii) region-specific report division,
that involves the annotation and categorization of radiology text reports by
the anatomical regions they refer; (iii) rule-based question generation, which
entails extracting entities from the sentence, and formulating visual question
answering (VQA) pairs linked to specific segmentation masks.

Figure 2: Overview of results obtained from each step of the data construction
pipeline.

#### Segmentation Mask Generation

To segment as many anatomical regions as possible, we employ the recent SAT
[13] model. It is a knowledge-enhanced segmentation model, that employs
natural language as prompts to effectively segment 3D medical volumes. The
model has been trained on 72 diverse segmentation datasets, covering 498
classes across various anatomical regions including the brain, head and neck,
thorax, spine, abdomen, and limbs. For our research, SAT is adopted to execute
detailed segmentation across all volumes of the CT-RATE dataset. Specifically,
we focus on segmenting 197 regions pertinent to chest CT scans, enabling
precise anatomical analysis. The list of segmented anatomies was organized
into a hierarchical tree as shown in Supplementary B. This includes several
major regions such as the lungs, trachea and bronchi, mediastinum, heart,
pleura, bones, thyroid, breasts, abdomen, and other areas.

#### Region-wise Reports Generation

The goal here is to break the entire reports into region-wise descriptions, we
observe that the OpenAI GPT-4 [2] model can complete such task in very high
accuracy [14, 15]. However, employing GPT-4 on the entire set would be
prohibitively expensive, thus, we train a model for such report division.
Specifically, we first employ GPT-4 [2] to annotate the anatomical regions of
each sentence in the “FINDINGS” section of 2,500 radiology reports, which
comprise all reports in the validation set. The prompt used is as follows.

You are a radiologist tasked with extracting anatomical regions from the
“FINDINGS” section of radiology reports. For each sentence provided, identify
the corresponding anatomical regions. Ensure each identified region is an
entry from a predefined list: [region_list]. If a sentence mentions “left” or
“right”, these qualifiers should precede the anatomical region (e.g., left
kidney). Given input in the format: <Input><findings><$\backslash$Input>.
Please reply in the following JSON format: {<sentence>: [region1,region2,…],
<sentence>: [region1]}.

This process results in 15,926 annotated sentences. Subsequently, we divide
these sentences into training and validation subsets in an 8:2 ratio and train
a GPT-2 model using the annotated sentence along with the two preceding and
following sentences from the report as input, if available. The model is
designed to output the list of anatomical regions associated with each
sentence. For instance, for the target sentence “No pleural effusion was
detected on the left.”, the input is “There is minimal pleural effusion on the
right. No pleural effusion was detected on the left. Atelectasis is observed
in the middle lobe and lower lobe of the right lung. A malignant mass is
observed around the lower lobe bronchi of the left lung.”, and the expected
output is “left lung”. The model achieves an accuracy of 94.56% on the
validation set. Consequently, we employ this model to perform inference on all
sentences across the entire dataset of reports. More examples of segmentation
results and structured reports can be seen in Supplementary Section A.

#### Named Entity Recognition

Through the abovementioned process, each “FINDINGS” is divided into multiple
sentences, and each is associated with one or multiple segmented regions. To
facilitate the generation of detailed question-answer pairs from these
sentences, we initially employ an in-house Named-Entity Recognition (NER)
model to analyze all sentences. This process involves extracting entities that
can be categorized into “anatomy”, “abnormality”, and “non-abnormality”. Here,
“anatomy” pertains to the anatomical regions, “abnormality” refers to findings
or diseases identified as present, and “non-abnormality” indicates findings or
diseases that are reported as absent. Subsequently, all extracted
"abnormality" and "non-abnormality" entities undergo quality evaluation using
GPT-4, allowing us to filter out and revise any inaccuracies. For instance, in
cases where the NER model extracts abnormalities such as “structural
distortion and volume loss”, GPT-4 will segment it into “structural
distortion” and “volume loss” for more accurate categorization. We filter out
abnormalities with a GPT-4 output of “no” and update them to the revised
versions provided by GPT-4. The prompt used is as follows.

You are a radiologist. You will receive an input that consists of a sentence
from the findings section of a chest CT report, followed by a phrase extracted
from that sentence. Assess whether the extracted phrase is an abnormality. The
abnormality should not be a size or a location. If the extracted phrase does
not accurately represent an abnormality, your output should provide the
correct abnormalities. The input format will be
<Input><sentence><sep><phrase></Input>. Your response should be in JSON
format, structured as follows: {abnormality: [abnormality1, abnormality2,…],
phrase: “yes” or “no”}. Note that the output should be in JSON format.

In addition, for the “IMPRESSION” section, we directly utilize GPT-4 to
extract all disorders mentioned, and detailed information regarding the
presence of any abnormalities in specific anatomical regions. The prompt used
is as follows.

You are a radiologist performing disorders summary from the “IMPRESSION”
section in the patient’s chest CT scan report. Given input in the format:
<Input><impression><$\backslash$Input> Please reply with the JSON format
following template: {disorders:[disorder1,disorder2,…]}

Level | Question type | Answer type | Example | Train | Validation
---|---|---|---|---|---
Region | Abnormality | Open | What abnormality is seen in the {region }? | 483,507 | 30,831
Region | Presence | Close | Is there evidence of {abnormality} in the {region }? | 556,050 | 34,785
Region | Location | Close & Open | Where in the image is the {abnormality} seen? | 285,111 | 17,750
Region | Size | Open | What is the size of the {abnormality} in the {region}? | 18,389 | 1,259
Case | Disorder | Open | What disorder can be identified from this volume? | 24,128 | 1,564
Table 1:  Question and answer types in the proposed datasets. The placeholders
{region } and {abnormality} in the question templates are dynamically replaced
with entities extracted from sentences. Figure 3: Partial visualization of the
anatomical hierarchical tree with abnormalities annotated.

#### Rule-based Question Generation

Here, we describe the procedure for generating grounded visual question-
answering (VQA) data based on “FINDINGS” extracted from the report section,
while case-level visual question-answering data is generated from the
“IMPRESSION” section. Tab. 1 presents the 5 question types in the proposed
dataset. The detailed rules will be introduced in the following sections.

First, after the region-wise report generation, the sentences in the findings
section can be classified as follows:

  * •

Normal Findings: Sentences that report no significant changes from normal
health conditions.

    * –

No abnormality entities in the sentence: Sentences that mention specific
anatomical regions without noting any abnormalities. For example, “Thoracic
aorta diameter is normal.”

    * –

No anatomy entities in the sentence: Sentences that solely note the absence of
specific abnormalities without referring to any anatomical regions. For
example, “No pleural effusion was detected.”

    * –

With anatomy and abnormality entities in the sentence: Sentences that
explicitly state the absence of abnormalities in specific anatomical regions.
For example, “Bilateral adrenal glands were normal and no space-occupying
lesion was detected.”

  * •

Abnormal Findings: Sentences that report differences from normal anatomical
conditions.

    * –

No anatomy entities in the sentence: Sentences that report an abnormal
finding, but do not specify an anatomical region. For example, “Mild hiatal
hernia is observed.”

    * –

With anatomy entities in the sentence: Sentences that include both an
anatomical reference and describe an abnormality. For example, “There is
narrowing of the spinal canal at the dorso-lumbar level.”

We then construct an anatomical disorder tree for each report, based on the
anatomical hierarchical tree introduced in Supplementary Section B. This
involves marking any abnormalities on the tree for all nodes within the
hierarchy if they are present. As shown in Fig. 3 This comprehensive data
enables us to transit to generating questions. Taking inspiration from the
previous research [16, 17], we categorize the questions into four types: 1)
abnormality, 2) presence, 3) location, 4) size. Tab. 1 shows examples of the
different question types. Note that we have designed 50 templates for each
question type. The details of all templates are provided in the supplementary
materials. For instance, when analyzing a sentence indicating normal findings,
such as “Bilateral adrenal glands were normal and no space-occupying lesion
was detected.”, questions can include “Is there any evidence of abnormality in
adrenal glands?” and “What abnormality is seen in the adrenal glands?”.
Conversely, for a sentence in abnormal findings, “There is a narrowing of the
spinal canal at the dorso-lumbar level.”, the question can be “Is there any
evidence of narrowing in the spinal canal?” “What abnormality is seen at the
dorso-lumbar level of the spinal canal?” and “Where in the spinal canal is the
narrowing located?”. For impression sentences, since we have already extracted
disorders, we can generate case-level questions such as “What disorder can be
identified from this volume?” for each case.

Figure 4: (a) Distribution of Normal vs. Abnormal Cases: Training set
analysis. (b) Distribution of Normal vs. Abnormal Cases: Validation set
analysis. (c) Abnormalities Word Cloud: Visual summary of frequent
abnormalities from findings. (d) Disorders Word Cloud: Visual summary of
frequent disorders from findings.

In summary, we have successfully generated 1.3M grounded Visual Question
Answering (VQA) pairs for training and 85k for validation, along with 24,128
case-level visual question answering pairs for training and 1,564 for
validation. Detailed counts for each type of VQA pair in both the training and
validation sets are provided in Table 1.

### Dataset Analysis

In this section, we analyze the abnormalities of the proposed dataset. The
proposed hierarchically structured abnormality tree for each case enables us
to systematically extract and analyze the abnormal findings. First, we
calculate the normal-to-abnormal case numbers for both the training set and
the validation set, as shown in Fig. 4. Second, based on the identified and
extracted abnormalities from all anatomical regions, we proceed to visualize
these findings using a word cloud. The word cloud of abnormalities is
presented in Fig. 4.

## Conclusion

In this paper, we develop an automated pipeline for generating grounded
datasets and introduce RadGenome-Chest CT, a comprehensive, large-scale,
region-guided 3D chest CT interpretation dataset based on CT-RATE. RadGenome-
Chest CTinclude 197 organ-level segmentation masks, 665 K multi-granularity
grounded reports, and 1.3 M grounded VQA pairs. We anticipate that RadGenome-
Chest CTwill significantly advance multimodal medical AI models, enabling them
to generate texts based on segmentation regions, thus enhancing
interpretability and patient care. We will release all segmentation masks,
grounded reports, and VQA pairs to support future research in this field.

## References

  * [1] Lu, M. Y. _et al._ Towards a visual-language foundation model for computational pathology.  _arXiv preprint arXiv:2307.12914_ (2023).
  * [2] Achiam, J. _et al._ Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * [3] Singhal, K. _et al._ Large language models encode clinical knowledge.  _Nature_ 620, 172–180 (2023).
  * [4] Team, G. _et al._ Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ (2023).
  * [5] Alayrac, J.-B. _et al._ Flamingo: a visual language model for few-shot learning.  _Advances in neural information processing systems_ 35, 23716–23736 (2022).
  * [6] Wu, C., Zhang, X., Zhang, Y., Wang, Y. & Xie, W.  Towards generalist foundation model for radiology.  _arXiv preprint arXiv:2308.02463_ (2023).
  * [7] Tu, T. _et al._ Towards generalist biomedical ai.  _NEJM AI_ 1, AIoa2300138 (2024).
  * [8] Li, C. _et al._ Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * [9] Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J.  A visual–language foundation model for pathology image analysis using medical twitter.  _Nature medicine_ 29, 2307–2316 (2023).
  * [10] Moor, M. _et al._ Med-flamingo: a multimodal medical few-shot learner.  In _Machine Learning for Health (ML4H)_ , 353–367 (PMLR, 2023).
  * [11] Johnson, A. E. _et al._ Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ 6, 317 (2019).
  * [12] Hamamci, I. E. _et al._ A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-shot detection of abnormalities.  _arXiv preprint arXiv:2403.17834_ (2024).
  * [13] Zhao, Z. _et al._ One model to rule them all: Towards universal segmentation for medical images with text prompts.  _arXiv preprint arXiv:2312.17183_ (2023).
  * [14] Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_ (2023).
  * [15] Liu, Q. _et al._ Exploring the boundaries of gpt-4 in radiology.  _arXiv preprint arXiv:2310.14573_ (2023).
  * [16] Hu, X. _et al._ Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering.  In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 4156–4165 (2023).
  * [17] Lau, J. J., Gayen, S., Ben Abacha, A. & Demner-Fushman, D.  A dataset of clinically generated visual questions and answers about radiology images.  _Scientific data_ 5, 1–10 (2018).

##  Appendix A Case Examples

Figure 5: Case examples of segmentation masks and structured reports.

##  Appendix B Hierarchy of anatomical regions

.1 Chest CT Volume. .2 Lung. .3 Lung. .4 Left Lung. .5 Left Lung Lower Lobe.
.5 Left Lung Upper Lobe. .4 Right Lung. .5 Right Lung Lower Lobe. .5 Right
Lung Middle Lobe. .5 Right Lung Upper Lobe. .4 Lung Lower Lobe. .5 Left Lung
Lower Lobe. .5 Right Lung Lower Lobe. .4 Lung Upper Lobe. .5 Left Lung Upper
Lobe. .5 Right Lung Upper Lobe. .2 Trachea and Bronchie. .3 Trachea. .3
Bronchie. .2 Mediastinum. .3 Brachiocephalic Trunk. .3 Brachiocephalic Vein.
.4 Left Brachiocephalic Vein. .4 Right Brachiocephalic Vein. .3 Superior Vena
Cava. .3 Aorta. .3 Pulmonary Artery. .3 Thymus. .3 Pulmonary Vein. .3
Mediastinal Tissue. .3 Subclavian Artery. .4 Left Subclavian Artery. .4 Right
Subclavian Artery. .2 Heart. .3 Heart. .4 Heart Atrium. .5 Left Heart Atrium.
.5 Right Heart Atrium. .4 Heart Ventricle. .5 Left Heart Ventricle. .5 Right
Heart Ventricle. .4 Heart Ascending Aorta. .4 Heart Tissue. .5 Myocardium. .4
Left Auricle of Heart. .2 Esophagus. .3 Esophagus. .4 Cervical Esophagus. .4
Cricopharyngeal Inlet. .2 Pleura. .2 Bone. .3 Bone. .4 Spinal Cord. .4 Spinal
Canal. .4 Vertebrae. .5 Cervical Vertebrae. .6 Cervical Vertebrae 1 (C1). .6
Cervical Vertebrae 2 (C2). .6 Cervical Vertebrae 3 (C3). .6 Cervical Vertebrae
4 (C4). .6 Cervical Vertebrae 5 (C5). .6 Cervical Vertebrae 6 (C6). .6
Cervical Vertebrae 7 (C7). .5 Thoracic Vertebrae. .6 Thoracic Vertebrae 1
(T1). .6 Thoracic Vertebrae 2 (T2). .6 Thoracic Vertebrae 3 (T3). .6 Thoracic
Vertebrae 4 (T4). .6 Thoracic Vertebrae 5 (T5). .6 Thoracic Vertebrae 6 (T6).
.6 Thoracic Vertebrae 7 (T7). .6 Thoracic Vertebrae 8 (T8). .6 Thoracic
Vertebrae 9 (T9). .6 Thoracic Vertebrae 10 (T10). .6 Thoracic Vertebrae 11
(T11). .6 Thoracic Vertebrae 12 (T12). .5 Lumbar Vertebrae. .6 Lumbar
Vertebrae 1 (L1). .6 Lumbar Vertebrae 2 (L2). .6 Lumbar Vertebrae 3 (L3). .6
Lumbar Vertebrae 4 (L4). .6 Lumbar Vertebrae 5 (L5). .6 Lumbar Vertebrae 6
(L6). .5 Sacral Vertebrae 1 (S1). .4 Clavicle. .5 Left Clavicle. .5 Right
Clavicle. .4 Scapula. .5 Left Scapula. .5 Right Scapula. .4 Humerus. .5 Left
Humerus. .5 Right Humerus. .4 Femur. .5 Left Femur. .5 Right Femur. .4 Head of
Femur. .5 Left Head of Femur. .5 Right Head of Femur. .4 Rib. .5 Left Rib. .6
Left Rib 1. .6 Left Rib 2. .6 Left Rib 3. .6 Left Rib 4. .6 Left Rib 5. .6
Left Rib 6. .6 Left Rib 7. .6 Left Rib 8. .6 Left Rib 9. .6 Left Rib 10. .6
Left Rib 11. .6 Left Rib 12. .5 Right Rib. .6 Right Rib 1. .6 Right Rib 2. .6
Right Rib 3. .6 Right Rib 4. .6 Right Rib 5. .6 Right Rib 6. .6 Right Rib 7.
.6 Right Rib 8. .6 Right Rib 9. .6 Right Rib 10. .6 Right Rib 11. .6 Right Rib
12. .4 Rib 1. .5 Left Rib 1. .5 Right Rib 1. .4 Rib 2. .5 Left Rib 2. .5 Right
Rib 2. .4 Rib 3. .5 Left Rib 3. .5 Right Rib 3. .4 Rib 4. .5 Left Rib 4. .5
Right Rib 4. .4 Rib 5. .5 Left Rib 5. .5 Right Rib 5. .4 Rib 6. .5 Left Rib 6.
.5 Right Rib 6. .4 Rib 7. .5 Left Rib 7. .5 Right Rib 7. .4 Rib 8. .5 Left Rib
8. .5 Right Rib 8. .4 Rib 9. .5 Left Rib 9. .5 Right Rib 9. .4 Rib 10. .5 Left
Rib 10. .5 Right Rib 10. .4 Rib 11. .5 Left Rib 11. .5 Right Rib 11. .4 Rib
12. .5 Left Rib 12. .5 Right Rib 12. .4 Rib Cartilage. .4 Costal Cartilage. .4
Sternum. .5 Manubrium of Sternum. .4 Eustachian Tube Bone. .5 Left Eustachian
Tube Bone. .5 Right Eustachian Tube Bone. .2 Thyroid. .3 Thyroid. .4 Thyroid
Gland. .5 Left Thyroid. .5 Right Thyroid. .2 Breast. .3 Left Breast. .3 Right
Breast. .2 Abdomen. .3 Abdomen. .4 Abdominal Tissue. .4 Adrenal Gland. .5 Left
Adrenal Gland. .5 Right Adrenal Gland. .4 Aorta. .4 Colon. .4 Duodenum. .4
Gallbladder. .4 Intestine. .5 Small Bowel. .4 Kidney. .5 Left Kidney. .5 Right
Kidney. .4 Liver. .5 Left Lobe of Liver. .6 Left Lateral Inferior Segment of
Liver. .6 Left Lateral Superior Segment of Liver. .6 Left Medial Segment of
Liver. .5 Right Lobe of Liver. .6 Right Anterior Inferior Segment of Liver. .6
Right Anterior Superior Segment of Liver. .6 Right Posterior Inferior Segment
of Liver. .6 Right Posterior Superior Segment of Liver. .4 Liver Vessel. .4
Caudate Lobe. .4 Pancreas. .4 Portal Vein and Splenic Vein. .4 Rectum. .4
Renal Artery. .4 Renal Vein. .4 Spleen. .4 Stomach. .4 Celiac Trunk. .2
Others. .3 Thoracic Cavity. .3 Prostate. .3 Urinary Bladder. .3 Carotid
Artery. .4 Common Carotid Artery. .4 Internal Carotid Artery. .4 Left Carotid
Artery. .5 Left Common Carotid Artery. .5 Left Internal Carotid Artery. .4
Right Carotid Artery. .5 Right Common Carotid Artery. .5 Right Internal
Carotid Artery. .3 Iliac Artery. .3 Iliac Vena. .3 Iliac Vein. .3 Left Iliac
Artery. .3 Left Iliac Vena. .3 Right Iliac Artery. .3 Right Iliac Vena. .3
Inferior Vena Cava. .3 Internal Jugular Vein. .3 Larynx. .4 Larynx Glottis. .4
Larynx Supraglottis. .3 Muscle.

##  Appendix C Question template

#### Abnormality

  1. 1.

What are the abnormalities in the {region}?

  2. 2.

What abnormalities are present in the {region}?

  3. 3.

What types of abnormality are visible in the {region}?

  4. 4.

What type of abnormality is visible in the {region}?

  5. 5.

What kind of abnormalities are observed in the {region}?

  6. 6.

What kinds of abnormalities can be identified in the {region}?

  7. 7.

What specific abnormalities are detected in the {region}?

  8. 8.

What specific types of abnormalities are evident in the {region}?

  9. 9.

What types of abnormalities are evident upon examination of the {region}?

  10. 10.

What types of abnormalities can be seen in the {region}?

  11. 11.

What are the anomalies in the {region}?

  12. 12.

What anomalies are present in the {region}?

  13. 13.

What types of anomalies are visible in the {region}?

  14. 14.

What type of anomalies is visible in the {region}?

  15. 15.

What kind of anomalies are observed in the {region}?

  16. 16.

What kinds of anomalies can be identified in the {region}?

  17. 17.

What specific anomalies are detected in the {region}?

  18. 18.

What specific types of anomalies are evident in the {region}?

  19. 19.

What types of anomalies are evident upon examination of the {region}?

  20. 20.

What types of anomalies can be seen in the {region}?

  21. 21.

What are the abnormal findings in the {region}?

  22. 22.

What abnormal findings are present in the {region}?

  23. 23.

What types of abnormal findings are visible in the {region}?

  24. 24.

What type of abnormal findings is visible in the {region}?

  25. 25.

What kind of abnormal findings are observed in the {region}?

  26. 26.

What kinds of abnormal findings can be identified in the {region}?

  27. 27.

What specific abnormal findings are detected in the {region}?

  28. 28.

What specific types of abnormal findings are evident in the {region}?

  29. 29.

What types of abnormal findings are evident upon examination of the {region}?

  30. 30.

What types of abnormal findings can be seen in the {region}?

  31. 31.

What are the irregular findings in the {region}?

  32. 32.

What irregular findings are present in the {region}?

  33. 33.

What types of irregular findings are visible in the {region}?

  34. 34.

What type of irregular findings is visible in the {region}?

  35. 35.

What kind of irregular findings are observed in the {region}?

  36. 36.

What kinds of irregular findings can be identified in the {region}?

  37. 37.

What specific irregular findings are detected in the {region}?

  38. 38.

What specific types of irregular findings are evident in the {region}?

  39. 39.

What types of irregular findings are evident upon examination of the {region}?

  40. 40.

What types of irregular findings can be seen in the {region}?

  41. 41.

What are the irregularities in the {region}?

  42. 42.

What irregularities are present in the {region}?

  43. 43.

What types of irregularities are visible in the {region}?

  44. 44.

What type of irregularities is visible in the {region}?

  45. 45.

What kind of irregularities are observed in the {region}?

  46. 46.

What kinds of irregularities can be identified in the {region}?

  47. 47.

What specific irregularities are detected in the {region}?

  48. 48.

What specific types of irregularities are evident in the {region}?

  49. 49.

What types of irregularities are evident upon examination of the {region}?

  50. 50.

What types of irregularities can be seen in the {region}?

#### Presence

  1. 1.

Can {abnormality} be identified in the {region}?

  2. 2.

Can {abnormality} be observed in the {region}?

  3. 3.

Can {abnormality} be detected in the {region}?

  4. 4.

Can {abnormality} be seen in the {region}?

  5. 5.

Can {abnormality} be founded in the {region}?

  6. 6.

Can {abnormality} be recognized in the {region}?

  7. 7.

Can we detect any signs of {abnormality} in the {region}?

  8. 8.

Can we observe any signs of {abnormality} in the {region}?

  9. 9.

Can we recognize any signs of {abnormality} in the {region}?

  10. 10.

Can we see {abnormality} in the {region}?

  11. 11.

Can we find {abnormality} in the {region}?

  12. 12.

Can we detect {abnormality} in the {region}?

  13. 13.

Can we observe {abnormality} in the {region}?

  14. 14.

Can we identify {abnormality} in the {region}?

  15. 15.

Is there any sign of {abnormality} in the {region}?

  16. 16.

Is there any indication of {abnormality} in the {region}?

  17. 17.

Is there any evidence of {abnormality} in the {region}?

  18. 18.

Is there any suggestion of {abnormality} in the {region}?

  19. 19.

Is there a clear sign of {abnormality} in the {region}?

  20. 20.

Is there a clear indication of {abnormality} in the {region}?

  21. 21.

Is there a clear evidence of {abnormality} in the {region}?

  22. 22.

Is there a clear suggestion of {abnormality} in the {region}?

  23. 23.

Is {abnormality} visibly present in the {region}?

  24. 24.

Is {abnormality} clearly visible in the {region}?

  25. 25.

Is there any visual evidence suggesting {abnormality} in the {region}?

  26. 26.

Is there any indication of {abnormality} upon examination of the {region}?

  27. 27.

Is there any indication of {abnormality} in the {region}?

  28. 28.

Is there visual evidence of {abnormality} in the {region} on this scan?

  29. 29.

Are there any visible indications of {abnormality} in this {region}?

  30. 30.

Are there any visible cues indicating {abnormality} in the {region}?

  31. 31.

Are there any visible indicators of {abnormality} in the {region}?

  32. 32.

Are there any clear indications of {abnormality} in this {region}?

  33. 33.

Are there any clear cues indicating {abnormality} in the {region}?

  34. 34.

Are there any clear indicators of {abnormality} in the {region}?

  35. 35.

Are there any indications of {abnormality} in this {region}?

  36. 36.

Are there any cues indicating {abnormality} in the {region}?

  37. 37.

Are there any indicators of {abnormality} in the {region}?

  38. 38.

Are there any observable signs of {abnormality} in the {region}?

  39. 39.

Are there any signs of {abnormality} in the {region}?

  40. 40.

Are there any features of {abnormality} in the {region}?

  41. 41.

Does the {region} show the presence of {abnormality}?

  42. 42.

Does the {region} show the existence of {abnormality}?

  43. 43.

Does the image suggest the presence of {abnormality} in the {region}?

  44. 44.

Does the image suggest the existence of {abnormality} in the {region}?

  45. 45.

Does the {region} exhibit any evidence of {abnormality}?

  46. 46.

Does the {region} display any features suggestive of {abnormality}?

  47. 47.

Does the {region} display any characteristics suggestive of {abnormality}?

  48. 48.

Does the {region} exhibit any characteristics indicative of {abnormality}?

  49. 49.

Does the {region} exhibit any features indicative of {abnormality}?

  50. 50.

Does the visual features suggest the presence of {abnormality} in the
{region}?

#### Location

  1. 1.

Where is the {abnormality} located in the image?

  2. 2.

Where can the {abnormality} be found within the image?

  3. 3.

Where in the image is the {abnormality} located?

  4. 4.

Where in the image is the {abnormality} localized?

  5. 5.

Where in the image can the {abnormality} be found?

  6. 6.

Where in the image does the {abnormality} appear?

  7. 7.

Where in the image does the {abnormality} locate?

  8. 8.

Where in the image does the {abnormality} locate?

  9. 9.

Where specifically within the image is the {abnormality} located?

  10. 10.

Where exactly within the image is the {abnormality} located?

  11. 11.

Where exactly is the {abnormality} located in the image?

  12. 12.

Where specifically is the {abnormality} located in the image?

  13. 13.

Where exactly within the image is the {abnormality} localized?

  14. 14.

Where specifically within the image is the {abnormality} localized?

  15. 15.

Where within the image can the {abnormality} be precisely located?

  16. 16.

Where exactly within the image does the {abnormality} present?

  17. 17.

Where within the image does the {abnormality} specifically present?

  18. 18.

Where in the image does the {abnormality} appear?

  19. 19.

What is the location of the {abnormality} in the image?

  20. 20.

What is the precise location of the {abnormality} in the image?

  21. 21.

What is the specific location of the {abnormality} within the image?

  22. 22.

What is the precise region of the {abnormality} in the image?

  23. 23.

What is the specific region of the {abnormality} within the image?

  24. 24.

What particular region within the image does the {abnormality} occupy?

  25. 25.

What particular location within the image does the {abnormality} occupy?

  26. 26.

What specific location within the image does the {abnormality} occupy?

  27. 27.

What specific region within the image does the {abnormality} occupy?

  28. 28.

What specific area of the image does the {abnormality} occupy?

  29. 29.

What specific region of the image does the {abnormality} appear?

  30. 30.

What specific spot within the image contains the {abnormality}?

  31. 31.

What particular region of the image is affected by the {abnormality}?

  32. 32.

What specific area within the image is impacted by the {abnormality}?

  33. 33.

What specific region within the image is impacted by the {abnormality}?

  34. 34.

What specific location within the image is impacted by the {abnormality}?

  35. 35.

What particular region within the image is affected by the {abnormality}?

  36. 36.

What particular area within the image is affected by the {abnormality}?

  37. 37.

What particular location within the image is affected by the {abnormality}?

  38. 38.

What specific region within the image does the {abnormality} affect?

  39. 39.

What specific area within the image does the {abnormality} affect?

  40. 40.

What specific location within the image does the {abnormality} affect?

  41. 41.

What specific location within the image does the {abnormality} appear?

  42. 42.

What specific region within the image does the {abnormality} appear?

  43. 43.

What specific area within the image does the {abnormality} appear?

  44. 44.

What particular spot within the image does the {abnormality} present?

  45. 45.

What particular area within the image does the {abnormality} present?

  46. 46.

What particular region within the image does the {abnormality} present?

  47. 47.

What particular location within the image does the {abnormality} present?

  48. 48.

What specific area within the image does the {abnormality} occur?

  49. 49.

What specific location within the image does the {abnormality} occur?

  50. 50.

What specific region within the image does the {abnormality} occur?

#### Size

  1. 1.

What is the approximate size of the {abnormality} in the {region}?

  2. 2.

What is the approximate scale of the {abnormality} in the {region}?

  3. 3.

What is the approximate size range of the {abnormality} in the {region}?

  4. 4.

What is the approximate magnitude of the {abnormality} in the {region}?

  5. 5.

What is the approximate dimension of the {abnormality} in the {region}?

  6. 6.

What is the approximate measurement of the {abnormality} in the {region}?

  7. 7.

What is the estimated size of the {abnormality} in the {region}?

  8. 8.

What is the estimated scale of the {abnormality} in the {region}

  9. 9.

What is the estimated size range of the {abnormality} in the {region}?

  10. 10.

What is the estimated magnitude of the {abnormality} in the {region}?

  11. 11.

What is the estimated dimension of the {abnormality} in the {region}?

  12. 12.

What is the estimated measurement of the {abnormality} in the {region}?

  13. 13.

What is the size assessment of the {abnormality} in the {region}?

  14. 14.

What is the measurement of the {abnormality} in the {region}?

  15. 15.

What is the scale of the {abnormality} in the {region}?

  16. 16.

What is the size of the {abnormality} in the {region}?

  17. 17.

What is the size range of the {abnormality} in the {region}?

  18. 18.

What is the magnitude of the {abnormality} in the {region}?

  19. 19.

What is the dimension of the {abnormality} in the {region}?

  20. 20.

What is the overall size of the {abnormality} in the {region}?

  21. 21.

What is the overall scale of the {abnormality} in the {region}?

  22. 22.

What is the overall measurement of the {abnormality} in the {region}?

  23. 23.

What is the overall size range of the {abnormality} in the {region}?

  24. 24.

What is the overall magnitude of the {abnormality} in the {region}?

  25. 25.

What is the overall dimension of the {abnormality} in the {region}?

  26. 26.

What is the scale of the {abnormality} detected in the {region}?

  27. 27.

What is the size of the {abnormality} detected in the {region}?

  28. 28.

What is the size range of the {abnormality} detected in the {region}?

  29. 29.

What is the measurement of the {abnormality} detected in the {region}?

  30. 30.

What is the magnitude of the {abnormality} detected in the {region}?

  31. 31.

What is the dimension of the {abnormality} detected in the {region}?

  32. 32.

What is the scale of the {abnormality} appeared in the {region}?

  33. 33.

What is the size of the {abnormality} appeared in the {region}?

  34. 34.

What is the size range of the {abnormality} appeared in the {region}?

  35. 35.

What is the magnitude of the {abnormality} appeared in the {region}?

  36. 36.

What is the dimension of the {abnormality} appeared in the {region}?

  37. 37.

What is the measurement of the {abnormality} appeared in the {region}?

  38. 38.

How large is the affected {abnormality} area in the {region}?

  39. 39.

How large is the observed {abnormality} area in the {region}?

  40. 40.

How large is the {abnormality} observed in the {region}?

  41. 41.

How large is the {abnormality} in the {region}?

  42. 42.

How large does the {abnormality} appear in the {region}?

  43. 43.

How large does the {abnormality} appear to be in the {region}?

  44. 44.

How large is the {abnormality} area in the {region}?

  45. 45.

How large is the affected {abnormality} area in the {region}?

  46. 46.

How big is the affected {abnormality} area in the {region}?

  47. 47.

How big is the observed {abnormality} area in the {region}?

  48. 48.

How big is the {abnormality} observed in the {region}?

  49. 49.

How big is the {abnormality} in the {region}?