* Becker and Quille (2019) Brett A. Becker and Keith Quille. 2019.  50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In _Proceedings of the 50th ACM Technical Symposium on Computer Science Education_ (Minneapolis, MN, USA) _(SIGCSE ’19)_. Association for Computing Machinery, New York, NY, USA, 338–344.  https://doi.org/10.1145/3287324.3287432
  * Bellettini et al. (2023) Carlo Bellettini, Michael Lodi, Violetta Lonati, Mattia Monga, and Anna Morpurgo. 2023.  Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. In _Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU_. INSTICC, SciTePress, 59–69.  https://doi.org/10.5220/0012007500003470
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 610–623.  https://doi.org/10.1145/3442188.3445922
  * Biggs (1996) John Biggs. 1996\.  Enhancing teaching through constructive alignment.  _Higher education_ 32, 3 (1996), 347–364.
  * Biggs and Collis (2014) John B Biggs and Kevin F Collis. 2014.  _Evaluating the Quality of Learning: The SOLO Raxonomy (Structure of the Observed Learning Outcome)_.  Academic Press.
  * Bird et al. (2023) Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023\.  Taking Flight with Copilot: Early Insights and Opportunities of AI-Powered Pair-Programming Tools.  _Queue_ 20, 6 (jan 2023), 35–57.  https://doi.org/10.1145/3582083
  * Birhane et al. (2022) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022\.  The Values Encoded in Machine Learning Research. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 173–184.  https://doi.org/10.1145/3531146.3533083
  * Bommasani et al. (2022) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022.  On the Opportunities and Risks of Foundation Models.  arXiv:2108.07258 [cs.LG]
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative Research in Psychology_ 3, 2 (2006), 77–101.  https://doi.org/10.1191/1478088706qp063oa
  * Brennan and Lesage (2022) Robert W Brennan and Jonathan Lesage. 2022.  Exploring the Implications of Openai Codex on Education for Industry 4.0. In _International Workshop on Service Orientation in Holonic and Multi-Agent Manufacturing_. Springer, 254–266.
  * Brown et al. (2018) Neil C. C. Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018.  Blackbox, Five Years On: An Evaluation of a Large-Scale Programming Data Collection Project. In _Proceedings of the 2018 ACM Conference on International Computing Education Research_ (Espoo, Finland) _(ICER ’18)_. Association for Computing Machinery, New York, NY, USA, 196–204.  https://doi.org/10.1145/3230977.3230991
  * Brown et al. (2014) Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014.  Blackbox: A Large Scale Repository of Novice Programmers’ Activity. In _Proceedings of the 45th ACM Technical Symposium on Computer Science Education_ (Atlanta, Georgia, USA) _(SIGCSE ’14)_. Association for Computing Machinery, New York, NY, USA, 223–228.  https://doi.org/10.1145/2538862.2538924
  * Brusilovsky et al. (2023a) Peter Brusilovsky, Barbara J Ericson, Cay S Horstmann, and Christian Servin. 2023a.  The Future of Computing Education Materials.  (2023).
  * Brusilovsky et al. (2023b) Peter Brusilovsky, Barbara J. Ericson, Cay S. Horstmann, Christian Servin, Frank Vahid, and Craig Zilles. 2023b.  The Future of Computing Education Materials.  https://csed.acm.org/wp-content/uploads/2023/03/Educational-Materials-First-Draft-1.pdf First Draft, to be published in the CS2023: ACM/IEEE- CS/AAAI Computer Science Curricula.
  * Bull and Kharrufa (2023) Christopher Bull and Ahmed Kharrufa. 2023.  Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it.  _IEEE Software_ (2023).
  * Chan (2023) Cecilia Ka Yuk Chan. 2023\.  A Comprehensive AI Policy Education Framework for University Teaching and Learning.  arXiv:2305.00280 [cs.CY]
  * Chan and Lee (2023) Cecilia Ka Yuk Chan and Katherine K. W. Lee. 2023.  The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and Millennial Generation teachers?  arXiv:2305.02878 [cs.CY]
  * Chan and Tsi (2023) Cecilia Ka Yuk Chan and Louisa H. Y. Tsi. 2023.  The AI Revolution in Education: Will AI Replace or Assist Teachers in Higher Education?  arXiv:2305.01185 [cs.CY]
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating Large Language Models Trained on Code.  https://arxiv.org/abs/2107.03374.
  * Cipriano and Alves (2023) Bruno Pereira Cipriano and Pedro Alves. 2023.  GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 61–67.  https://doi.org/10.1145/3587102.3588814
  * Clarke and Braun (2021) Victoria Clarke and Virginia Braun. 2021.  Thematic analysis: a practical guide.  _Thematic Analysis_ (2021), 1–100.
  * Clear et al. (2020) Alison Clear, Allen Parrish, John Impagliazzo, Pearl Wang, Paolo Ciancarini, Ernesto Cuadros-Vargas, Stephen Frezza, Judith Gal-Ezer, Arnold Pears, Shingo Takada, Heikki Topi, Gerrit van der Veer, Abhijat Vichare, Les Waguespack, and Ming Zhang. 2020.  _Computing Curricula 2020 Paradigms for Global Computing Education_.  ACM, New York.
  * Cooper et al. (2003) Stephen Cooper, Wanda Dann, and Randy Pausch. 2003\.  Teaching Objects-First in Introductory Computer Science. In _Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education_ (Reno, Navada, USA) _(SIGCSE ’03)_. Association for Computing Machinery, New York, NY, USA, 191–195.  https://doi.org/10.1145/611892.611966
  * Dakhel et al. (2023) Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jiang. 2023.  Github copilot ai pair programmer: Asset or liability?  _Journal of Systems and Software_ (2023), 111734.
  * Dakhel et al. (2022) Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2022.  GitHub Copilot AI Pair Programmer: Asset or Liability?  https://doi.org/10.48550/arXiv.2206.15331 arXiv:2206.15331 [cs]
  * Dale (2006) Nell B. Dale. 2006\.  Most Difficult Topics in CS1: Results of an Online Survey of Educators.  _SIGCSE Bull._ 38, 2 (jun 2006), 49–53.  https://doi.org/10.1145/1138403.1138432
  * de Freitas et al. (2023) Adrian de Freitas, Joel Coffman, Michelle de Freitas, Justin Wilson, and Troy Weingart. 2023\.  FalconCode: A Multiyear Dataset of Python Code Samples from an Introductory Computer Science Course. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 938–944.  https://doi.org/10.1145/3545945.3569822
  * DeepLearning.AI (2023) DeepLearning.AI. 2023\.  Generative AI with LLMs.  https://www.deeplearning.ai/courses/generative-ai-with-llms/
  * Demidova et al. (2023) Liliya A Demidova, Elena G Andrianova, Peter N Sovietov, and Artyom V Gorchakov. 2023.  Dataset of Program Source Codes Solving Unique Programming Exercises Generated by Digital Teaching Assistant.  _Data_ 8, 6 (2023), 109.
  * Denny et al. (2019a) Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Banaszkiewicz. 2019a.  Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer. In _Proceedings of the 2019 ACM Conference on International Computing Education Research_ (Toronto ON, Canada) _(ICER ’19)_. Association for Computing Machinery, New York, NY, USA, 259–267.  https://doi.org/10.1145/3291279.3339402
  * Denny et al. (2023a) Paul Denny, Brett A. Becker, Juho Leinonen, and James Prather. 2023a.  Chat Overflow: Artificially Intelligent Models for Computing Education - RenAIssance or ApocAIypse?. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. ACM, NY, NY, USA, 3–4.  https://doi.org/10.1145/3587102.3588773 Video: www.youtube.com/watch?v=KwVcRXQc3IU.
  * Denny et al. (2023b) Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. 2023b.  Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources.  arXiv:2306.10509 [cs.HC]
  * Denny et al. (2023c) Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023c.  Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1136–1142.  https://doi.org/10.1145/3545945.3569823
  * Denny et al. (2023d) Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023d.  Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators.  arXiv:2307.16364 [cs.HC]
  * Denny et al. (2019b) Paul Denny, James Prather, Brett A. Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit. 2019b.  A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. In _Proceedings of the 19th Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’19)_. Association for Computing Machinery, New York, NY, USA, Article 11, 10 pages.  https://doi.org/10.1145/3364510.3366170
  * Denny et al. (2023e) Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023e.  Computing Education in the Era of Generative AI.  arXiv:2306.02608 [cs.CY]
  * Denny et al. (2022) Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. 2022\.  Robosourcing Educational Resources – Leveraging Large Language Models for Learnersourcing.  https://doi.org/10.48550/ARXIV.2211.04715
  * D’ignazio and Klein (2020) Catherine D’ignazio and Lauren F Klein. 2020.  _Data Feminism_.  MIT press.
  * Dobslaw and Bergh (2023) Felix Dobslaw and Peter Bergh. 2023.  Experiences with Remote Examination Formats in Light of GPT-4.  _arXiv preprint arXiv:2305.02198_ (2023).
  * Doebling and Kazerouni (2021) Augie Doebling and Ayaan M. Kazerouni. 2021.  Patterns of Academic Help-Seeking in Undergraduate Computing Students. In _Proceedings of the 21st Koli Calling International Conference on Computing Education Research_ (Joensuu, Finland) _(Koli Calling ’21)_. Association for Computing Machinery, New York, NY, USA, Article 13, 10 pages.  https://doi.org/10.1145/3488042.3488052
  * Dohmke et al. (2023) Thomas Dohmke, Marco Iansiti, and Greg Richards. 2023\.  Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle.  arXiv:2306.15033 [econ.GN]
  * Druga and Otero (2023) Stefania Druga and Nancy Otero. 2023.  Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for Families.  arXiv:2305.10417 [cs.HC]
  * Elarde and Chong (2011) Joseph V. Elarde and Fatt-Fei Chong. 2011.  Introductory Computing Course Content: Educator and Student Perspectives. In _Proceedings of the 2011 Conference on Information Technology Education_ (West Point, New York, USA) _(SIGITE ’11)_. Association for Computing Machinery, New York, NY, USA, 55–60.  https://doi.org/10.1145/2047594.2047610
  * Ernst and Bavota (2022) Neil A. Ernst and Gabriele Bavota. 2022.  AI-Driven Development Is Here: Should You Worry?  _IEEE Software_ 39, 2 (mar 2022), 106–110.  https://doi.org/10.1109/ms.2021.3133805
  * Finnie-Ansley et al. (2022a) James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022a.  The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming _(ACE ’22)_. ACM, Online, 10–19.  https://doi.org/10.1145/3511861.3511863
  * Finnie-Ansley et al. (2022b) James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022b.  The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In _Proceedings of the 24th Australasian Computing Education Conference_ (Virtual Event, Australia) _(ACE ’22)_. Association for Computing Machinery, New York, NY, USA, 10–19.  https://doi.org/10.1145/3511861.3511863
  * Finnie-Ansley et al. (2023) James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023.  My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises _(ACE ’23)_. ACM, NY, NY, USA, 97–104.  https://doi.org/10.1145/3576123.3576134
  * Fisler (2014) Kathi Fisler. 2014\.  The Recurring Rainfall Problem _(ICER ’14)_. ACM, NY, NY, USA, 35–42.  https://doi.org/10.1145/2632320.2632346
  * Forman et al. (2023) Norbert Forman, József Udvaros, and Mihály Szilárd Avornicului. 2023\.  ChatGPT: A new study tool shaping the future for high school students.  _International Journal of Advanced Natural Sciences and Engineering Researches_ 7, 4 (May 2023), 95–102.  https://doi.org/10.59287/ijanser.562
  * French et al. (2023) Fiona French, David Levi, Csaba Maczo, Aiste Simonaityte, Stefanos Triantafyllidis, and Gergo Varda. 2023.  Creative Use of OpenAI in Education: Case Studies from Game Development.  _Multimodal Technologies and Interaction_ 7, 8 (2023).  https://doi.org/10.3390/mti7080081
  * Gherciu (2022) Pavel Gherciu. 2022\.  Net impact of large language models trained on code. In _Conferinţa tehnico-ştiinţifică a studenţilor, masteranzilor şi doctoranzilor_ , Vol. 1. 189–192.
  * Gupta et al. (2017) Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017\.  Deepfix: Fixing common c language errors by deep learning. In _Proceedings of the aaai conference on artificial intelligence_ , Vol. 31.
  * Hellas et al. (2018) Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V. Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018\.  Predicting Academic Performance: A Systematic Literature Review. In _Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education_ (Larnaca, Cyprus) _(ITiCSE 2018 Companion)_. Association for Computing Machinery, New York, NY, USA, 175–199.  https://doi.org/10.1145/3293881.3295783
  * Hellas et al. (2023) Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023.  Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests.  _arXiv preprint arXiv:2306.05715_ (2023).
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Coding Challenge Competence With APPS.  _NeurIPS_ (2021).
  * Idialu et al. ([n.d.]) Joy Idialu, Deborah Etsenake, and Norhan Abbas. [n.d.].  Whodunnit: Human or AI?  ([n. d.]).
  * Isserman (2003) Maurice Isserman. 2003\.  Plagiarism: A Lie of the Mind.  _The Chronicle Review_ 49 (2003).  Issue 34. http://chronicle.com
  * Jaipersaud et al. (2023) Brandon Jaipersaud, Paul Zhang, Jimmy Ba, Andrew Petersen, Lisa Zhang, and Michael R. Zhang. 2023\.  Decomposed Prompting to Answer Questions on a Course Discussion Board. In _Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky_ , Ning Wang, Genaro Rebolledo-Mendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 218–223.
  * Jakesch et al. (2023) Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023.  Co-Writing with Opinionated Language Models Affects Users’ Views. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 111, 15 pages.  https://doi.org/10.1145/3544548.3581196
  * Jalil et al. (2023) Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023.  ChatGPT and Software Testing Education: Promises & Perils. In _2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. 4130–4137.  https://doi.org/10.1109/ICSTW58534.2023.00078
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of Hallucination in Natural Language Generation.  _ACM Comput. Surv._ 55, 12 (2023), 248:1–248:38.  https://doi.org/10.1145/3571730
  * Johnson and Verdicchio (2023) Deborah G Johnson and Mario Verdicchio. 2023.  Ethical AI is not about AI.  _Commun. ACM_ 66, 2 (2023), 32–34.
  * Jones et al. (2008) Karl O. Jones, Reid Juliet, and Bartlett Rebecca. 2008\.  Cyber Cheating in an Information Technology Age.  _Digithum_ 10 (Dec. 2008).  https://raco.cat/index.php/Digithum/article/view/394993
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.  https://doi.org/10.1016/j.lindif.2023.102274
  * Kazemitabaar et al. (2023) Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023.  Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 455, 23 pages.  https://doi.org/10.1145/3544548.3580919
  * Kendon et al. (2023) Tyson Kendon, Leanne Wu, and John Aycock. 2023.  AI-Generated Code Not Considered Harmful. In _Proceedings of the 25th Western Canadian Conference on Computing Education_ (Vancouver, BC, Canada) _(WCCCE ’23)_. Association for Computing Machinery, New York, NY, USA, Article 3, 7 pages.  https://doi.org/10.1145/3593342.3593349
  * Khan and Madden (2018) Arshia Khan and Janna Madden. 2018.  Active learning: a new assessment model that boost confidence and learning while reducing test anxiety.  _International Journal of Modern Education and Computer Science_ 10, 12 (2018), 1.
  * Kiesler (2020a) Natalie Kiesler. 2020a.  On Programming Competence and Its Classification. In _Proceedings of the 20th Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’20)_. Association for Computing Machinery, New York, Article 1, 10 pages.  https://doi.org/10.1145/3428029.3428030
  * Kiesler (2020b) Natalie Kiesler. 2020b.  Towards a Competence Model for the Novice Programmer Using Bloom’s Revised Taxonomy - An Empirical Approach. In _Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education_ (Trondheim, Norway) _(ITiCSE ’20)_. Association for Computing Machinery, New York, NY, USA, 459–465.  https://doi.org/10.1145/3341525.3387419
  * Kiesler (2022) Natalie Kiesler. 2022\.  _Kompetenzförderung in der Programmierausbildung durch Modellierung von Kompetenzen und informativem Feedback_.  Dissertation. Johann Wolfgang Goethe-Universität, Frankfurt am Main.  Fachbereich Informatik und Mathematik.
  * Kiesler et al. (2023) Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023\.  Exploring the Potential of Large Language Models to Generate Formative Programming Feedback.  https://doi.org/10.48550/arXiv.2309.00029 arXiv:2309.00029 [cs.AI]
  * Kiesler and Schiffner (2022) Natalie Kiesler and Daniel Schiffner. 2022.  On the Lack of Recognition of Software Artifacts and IT Infrastructure in Educational Technology Research. In _20\. Fachtagung Bildungstechnologien (DELFI)_ , Peter A. Henning, Michael Striewe, and Matthias Wölfel (Eds.). Gesellschaft für Informatik e.V., Bonn, 201–206.  https://doi.org/10.18420/delfi2022-034
  * Kiesler and Schiffner (2023a) Natalie Kiesler and Daniel Schiffner. 2023a.  Large Language Models in Introductory Programming Education: ChatGPT’s Performance and Implications for Assessments. In _CoRR abs/2308.08572_.  https://doi.org/10.48550/arXiv.2308.08572 arXiv:2308.08572 [cs.SE]
  * Kiesler and Schiffner (2023b) Natalie Kiesler and Daniel Schiffner. 2023b.  Why We Need Open Data in Computer Science Education Research. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education Vol. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3587102.3588860
  * Kulal et al. (2019) Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019.  Spoc: Search-based pseudocode to code.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Latulipe et al. (2015) Celine Latulipe, N Bruce Long, and Carlos E Seminario. 2015\.  Structuring flipped classes with lightweight teams and gamification. In _Proceedings of the 46th ACM Technical Symposium on Computer Science Education_. 392–397.
  * Lau and Guo (2023) Sam Lau and Philip J. Guo. 2023.  From ”Ban It TillWe Understand It” to ”Resistance is Futile”: How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot _(ICER ’23)_. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3568813.3600138
  * Lee et al. (2022) Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022.  CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course.  _arXiv preprint arXiv:2210.14494_ (2022).
  * Leinonen et al. (2023a) Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023a.  Comparing Code Explanations Created by Students and Large Language Models.  arXiv:2304.03938 [cs.CY]
  * Leinonen et al. (2023b) Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023b.  Using Large Language Models to Enhance Programming Error Messages _(SIGCSE 2023)_. ACM, NY, NY, USA, 563–569.  https://doi.org/10.1145/3545945.3569770
  * Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, et al. 2022a.  Competition-Level Code Generation with AlphaCode.  https://doi.org/10.48550/ARXIV.2203.07814
  * Li et al. (2022b) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022b.  Competition-level code generation with alphacode.  _Science_ 378, 6624 (2022), 1092–1097.
  * Liffiton et al. (2023) Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023.  CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes.  arXiv:2308.06921 [cs.CY]
  * Lister et al. (2004) Raymond Lister, Elizabeth S. Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, Beth Simon, and Lynda Thomas. 2004.  A Multi-National Study of Reading and Tracing Skills in Novice Programmers.  _SIGCSE Bull._ 36, 4 (jun 2004), 119–150.  https://doi.org/10.1145/1041624.1041673
  * Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023\.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _arXiv preprint arXiv:2305.01210_ (2023).
  * Loksa et al. (2022) Dastyni Loksa, Lauren Margulieux, Brett A. Becker, Michelle Craig, Paul Denny, Raymond Pettit, and James Prather. 2022.  Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use.  _ACM Trans. Comput. Educ._ 22, 4, Article 39 (sep 2022), 31 pages.  https://doi.org/10.1145/3487050
  * Luxton-Reilly et al. (2018) Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018\.  Introductory Programming: A Systematic Literature Review. In _Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education_ (Larnaca, Cyprus) _(ITiCSE 2018 Companion)_. Association for Computing Machinery, New York, NY, USA, 55–106.  https://doi.org/10.1145/3293881.3295779
  * Ma et al. (2023) Qianou Ma, Tongshuang Wu, and Kenneth Koedinger. 2023.  Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming.  _arXiv preprint arXiv:2306.05153_ (2023).
  * MacNeil et al. (2023a) Stephen MacNeil, Joanne Kim, Juho Leinonen, Paul Denny, Seth Bernstein, Brett A. Becker, Michel Wermelinger, Arto Hellas, Andrew Tran, Sami Sarsa, James Prather, and Viraj Kumar. 2023a.  The Implications of Large Language Models for CS Teachers and Students. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1255.  https://doi.org/10.1145/3545947.3573358
  * MacNeil et al. (2016) Stephen MacNeil, Celine Latulipe, Bruce Long, and Aman Yadav. 2016\.  Exploring lightweight teams in a distributed learning environment. In _Proceedings of the 47th ACM Technical Symposium on Computing Science Education_. 193–198.
  * MacNeil et al. (2023b) Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023b.  Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 931–937.  https://doi.org/10.1145/3545945.3569785
  * MacNeil et al. (2023c) Stephen MacNeil, Andrew Tran, Juho Leinonen, Paul Denny, Joanne Kim, Arto Hellas, Seth Bernstein, and Sami Sarsa. 2023c.  Automatically Generating CS Learning Materials with Large Language Models. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1176.  https://doi.org/10.1145/3545947.3569630
  * Matelsky et al. (2023) Jordan K. Matelsky, Felipe Parodi, Tony Liu, Richard D. Lange, and Konrad P. Kording. 2023.  A large language model-assisted education tool to provide feedback on open-ended responses.  arXiv:2308.02439 [cs.CY]
  * Mittelstadt et al. (2016) Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016\.  The ethics of algorithms: Mapping the debate.  _Big Data & Society_ 3, 2 (2016), 2053951716679679\.  https://doi.org/10.1177/2053951716679679 arXiv:https://doi.org/10.1177/2053951716679679
  * Nam et al. (2023) Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2023.  In-IDE Generation-based Information Support with a Large Language Model.  arXiv:2307.08177 [cs.SE]
  * Nolan (2023) Beatrice Nolan. 2023\.  Here are the Schools and Colleges That Have Banned the Use of ChatGPT over Plagiarism and Misinformation Fears.  https://www.businessinsider.com/chatgpt-schools-colleges-ban-plagiarism-misinformation-education-2023-1
  * on Publication Ethics ([n.d.]) Committee on Publication Ethics. [n.d.].  Authorship and AI tools.  https://publicationethics.org/cope-position-statements/ai-author Accessed: 08 July 2023.
  * Orenstrakh et al. (2023) Michael Sheinman Orenstrakh, Oscar Karnalim, Carlos Anibal Suarez, and Michael Liut. 2023.  Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases.  arXiv:2307.07411 [cs.CL]
  * Pankiewicz and Baker (2023) Maciej Pankiewicz and Ryan S. Baker. 2023.  Large Language Models (GPT) for automating feedback on programming assignments.  arXiv:2307.00150 [cs.HC]
  * Patton (2002) Michael Quinn Patton. 2002\.  _Qualitative Research & Evaluation Methods_.  Sage, Thousand Oaks.
  * Pausch et al. (1995) Randy Pausch, Tommy Burnette, AC Capeheart, Matthew Conway, Dennis Cosgrove, Rob DeLine, Jim Durbin, Rich Gossweiler, Shuichi Koga, and Jeff White. 1995\.  Alice: Rapid prototyping system for virtual reality.  _IEEE Computer Graphics and Applications_ 15, 3 (1995), 8–11.
  * Petersen-Frey et al. (2022) Fynn Petersen-Frey, Marcus Soll, Louis Kobras, Melf Johannsen, Peter Kling, and Chris Biemann. 2022\.  Dataset of Student Solutions to Algorithm and Data Structure Programming Assignments. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_. 956–962.
  * Philbin (2023) Carrie Anne Philbin. 2023\.  Exploring the Potential of Artificial Intelligence Program Generators in Computer Programming Education for Students.  _ACM Inroads_ 14, 3 (aug 2023), 30–38.  https://doi.org/10.1145/3610406
  * Phung et al. (2023a) Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023a.  Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models.  arXiv:2302.04662 [cs.PL]
  * Phung et al. (2023b) Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023b.  Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors.  _International Journal of Management_ 21, 2 (2023), 100790\.
  * Piccolo et al. (2023) Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023\.  Many bioinformatics programming tasks can be automated with ChatGPT.  _arXiv preprint arXiv:2303.13528_ (2023).
  * Poldrack et al. (2023) Russell A Poldrack, Thomas Lu, and Gašper Beguš. 2023\.  AI-assisted coding: Experiments with GPT-4.  arXiv:2304.13187 [cs.AI]
  * Pollock (2019) Lori Pollock. 2019\.  A Collaborative Practicum Targeting Communication Skills for Computer Science Researchers. In _Proceedings of the 50th ACM Technical Symposium on Computer Science Education_ (Minneapolis, MN, USA) _(SIGCSE ’19)_. Association for Computing Machinery, New York, NY, USA, 845–851.  https://doi.org/10.1145/3287324.3287454
  * Porter and Zingaro (2023) Leo Porter and Daniel Zingaro. 2023.  _Learn AI-Assisted Python Programming With GitHub Copilot and ChatGPT_.  Manning, Shelter Island, NY, USA.  https://www.manning.com/books/learn-ai-assisted-python-programming
  * Prather et al. (2020) James Prather, Brett A. Becker, Michelle Craig, Paul Denny, Dastyni Loksa, and Lauren Margulieux. 2020\.  What Do We Think We Think We Are Doing? Metacognition and Self-Regulation in Programming _(ICER ’20)_. ACM, NY, NY, USA, 2–13.  https://doi.org/10.1145/3372782.3406263
  * Prather et al. (2018) James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018\.  Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In _Proceedings of the 2018 ACM Conference on International Computing Education Research_ (Espoo, Finland) _(ICER ’18)_. Association for Computing Machinery, New York, NY, USA, 41–50.  https://doi.org/10.1145/3230977.3230981
  * Prather et al. (2023) James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023\.  “It’s Weird That It Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers.  _ACM Trans. Comput.-Hum. Interact._ (aug 2023).  https://doi.org/10.1145/3617367 Just Accepted.
  * Puryear and Sprint (2022) Ben Puryear and Gina Sprint. 2022.  Github Copilot in the Classroom: Learning to Code with AI Assistance.  _J. Comput. Sci. Coll._ 38, 1 (nov 2022), 37–47.
  * Pădurean et al. (2023) Victor-Alexandru Pădurean, Georgios Tzannetos, and Adish Singla. 2023\.  Neural Task Synthesis for Visual Programming.  arXiv:2305.18342 [cs.LG]
  * Qadir (2023) Junaid Qadir. 2023\.  Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. In _2023 IEEE Global Engineering Education Conference (EDUCON)_. 1–9.  https://doi.org/10.1109/EDUCON54358.2023.10125121
  * Raj et al. (2021) Rajendra Raj, Mihaela Sabin, John Impagliazzo, David Bowers, Mats Daniels, Felienne Hermans, Natalie Kiesler, Amruth N. Kumar, Bonnie MacKellar, Renée McCauley, Syed Waqar Nabi, and Michael Oudshoorn. 2021.  Professional Competencies in Computing Education: Pedagogies and Assessment. In _Proceedings of the 2021 Working Group Reports on Innovation and Technology in Computer Science Education_ (Virtual Event, Germany) _(ITiCSE-WGR ’21)_. Association for Computing Machinery, New York, NY, USA, 133–161.  https://doi.org/10.1145/3502870.3506570
  * Rajabi et al. (2023) Parsa Rajabi, Parnian Taghipour, Diana Cukierman, and Tenzin Doleck. 2023.  Exploring ChatGPT’s impact on post-secondary education: A qualitative study. In _Western Canadian Conference on Computing Education (WCCCE’23), May 04–05, 2023_. Simon Fraser University.
  * Raman and Kumar (2022) Arun Raman and Viraj Kumar. 2022.  Programming Pedagogy and Assessment in the Era of AI/ML: A Position Paper. In _Proceedings of the 15th Annual ACM India Compute Conference_ (Jaipur, India) _(COMPUTE ’22)_. Association for Computing Machinery, New York, NY, USA, 29–34.  https://doi.org/10.1145/3561833.3561843
  * Raman et al. (2023) Raghu Raman, Santanu Mandal, Payel Das, Tavleen Kaur, Sanjanasri JP, and Prema Nedungadi. 2023\.  University students as early adopters of ChatGPT: Innovation Diffusion Study.  arXiv:10.21203/rs.3.rs-2734142/v1 [cs.HC] https://doi.org/10.21203/rs.3.rs-2734142/v1
  * Reeves et al. (2023) Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023.  Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 299–305.  https://doi.org/10.1145/3587102.3588805
  * Resnick et al. (2009) Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and Yasmin Kafai. 2009\.  Scratch: Programming for all.  _Commun. ACM_ 52 (11 2009), 60–67.  Issue 11. https://doi.org/10.1145/1592761.1592779
  * Ross et al. (2023) Steven I Ross, Michael Muller, Fernando Martinez, Stephanie Houde, and Justin D Weisz. 2023\.  A Case Study in Engineering a Conversational Programming Assistant’s Persona. In _Joint Proceedings of the ACM IUI Workshops 2023, March 2023, Sydney, Australia_.
  * Ryan and Bernard (2003) Gery W. Ryan and H. Russell Bernard. 2003.  Techniques to Identify Themes.  _Field Methods_ 15, 1 (2003), 85–109.  https://doi.org/10.1177/1525822X02239569 arXiv:https://doi.org/10.1177/1525822X02239569
  * Saito-Stehberger (2022) Dana Saito-Stehberger. 2022\.  Examples of Culturally Responsive Teaching in Computational Thinking Curriculum. In _Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2_ (Providence, RI, USA) _(SIGCSE 2022)_. Association for Computing Machinery, New York, NY, USA, 1055.  https://doi.org/10.1145/3478432.3499247
  * Samuelson (2020) Pamela Samuelson. 2020\.  AI Authorship?  _Commun. ACM_ 63, 7 (jun 2020), 20–22.  https://doi.org/10.1145/3401718
  * Sandoval et al. (2023) Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023.  Lost at C: A user study on the security implications of large language model code assistants.  _arXiv preprint arXiv:2208.09727_ (2023).
  * Sarsa et al. (2022) Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022\.  Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In _Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1_ (Lugano and Virtual Event, Switzerland) _(ICER ’22)_. ACM, NY NY, USA, 27–43.  https://doi.org/10.1145/3501385.3543957
  * Savelka et al. (2023a) Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. 2023a.  Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses.  _arXiv preprint arXiv:2306.10073_ (2023).
  * Savelka et al. (2023b) Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023b.  Large language models (gpt) struggle to answer multiple-choice questions about code.  _arXiv preprint arXiv:2303.08033_ (2023).
  * Savelka et al. (2023c) Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023c.  Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?  _arXiv preprint arXiv:2303.09325_ (2023).
  * Schulte and Bennedsen (2006) Carsten Schulte and Jens Bennedsen. 2006.  What Do Teachers Teach in Introductory Programming?. In _Proceedings of the Second International Workshop on Computing Education Research_ (Canterbury, United Kingdom) _(ICER ’06)_. Association for Computing Machinery, New York, NY, USA, 17–28.  https://doi.org/10.1145/1151588.1151593
  * Seraj et al. (2019) Mazyar Seraj, Eva-Sophie Katterfeldt, Kerstin Bub, Serge Autexier, and Rolf Drechsler. 2019\.  Scratch and Google Blockly: How Girls’ Programming Skills and Attitudes Are Influenced. In _Proceedings of the 19th Koli Calling International Conference on Computing Education Research_. ACM, New York.  https://doi.org/10.1145/3364510.3364515
  * Sheard et al. (2008) Judy Sheard, Angela Carbone, Raymond Lister, Beth Simon, Errol Thompson, and Jacqueline L. Whalley. 2008.  Going SOLO to Assess Novice Programmers.  _SIGCSE Bull._ 40, 3 (jun 2008), 209–213.  https://doi.org/10.1145/1597849.1384328
  * Sheard et al. (2017) Judy Sheard, Simon, Matthew Butler, Katrina Falkner, Michael Morgan, and Amali Weerasinghe. 2017\.  Strategies for Maintaining Academic Integrity in First-Year Computing Courses _(ITiCSE ’17)_. ACM, NY, NY, USA, 244–249.  https://doi.org/10.1145/3059009.3059064
  * Shulman (2005) Lee S. Shulman. 2005\.  Signature Pedagogies in the Professions.  _Daedalus_ 134, 3 (2005), 52–59.  http://www.jstor.org/stable/20027998
  * Simon et al. (2018) Simon, Judy Sheard, Michael Morgan, Andrew Petersen, Amber Settle, and Jane Sinclair. 2018\.  Informing Students about Academic Integrity in Programming. In _Proceedings of the 20th Australasian Computing Education Conference_ (Brisbane, Queensland, Australia) _(ACE ’18)_. Association for Computing Machinery, New York, NY, USA, 113–122.  https://doi.org/10.1145/3160489.3160502
  * Singla (2023) Adish Singla. 2023\.  Evaluating ChatGPT and GPT-4 for Visual Programming.  arXiv:2308.02522 [cs.LG]
  * Sok and Heng (2023) Sarin Sok and Kimkong Heng. 2023.  ChatGPT for Education and Research: A Review of Benefits and Risks.  arXiv:10.2139/ssrn.4378735 [cs.LG] http://dx.doi.org/10.2139/ssrn.4378735
  * Sridhar et al. (2023) Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Christopher Bogart, Jaromir Savelka, and Majd Sakr. 2023\.  Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives.  _arXiv preprint arXiv:2306.17459_ (2023).
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models _(CHI ’22)_. ACM, NY NY, USA, 1–7.
  * Wang et al. (2023a) Tianjia Wang, Daniel Vargas Dıaz, Chris Brown, and Yan Chen. 2023a.  Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives. In _2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_.
  * Wang et al. (2023b) Tianjia Wang, Daniel Vargas-Diaz, Chris Brown, and Yan Chen. 2023b.  Towards Adapting Computer Science Courses to AI Assistants’ Capabilities.  _arXiv preprint arXiv:2306.03289_ (2023).
  * Wermelinger (2023) Michel Wermelinger. 2023\.  Using GitHub Copilot to Solve Simple Programming Problems. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 172–178.  https://doi.org/10.1145/3545945.3569830
  * Widjojo and Treude (2023) Patricia Widjojo and Christoph Treude. 2023.  Addressing Compiler Errors: Stack Overflow or Large Language Models?  arXiv:2307.10793 [cs.SE]
  * Willis et al. (2020) Alistair Willis, Patricia Charlton, and Tony Hirst. 2020\.  Developing Students’ Written Communication Skills with Jupyter Notebooks. In _Proceedings of the 51st ACM Technical Symposium on Computer Science Education_ (Portland, OR, USA) _(SIGCSE ’20)_. Association for Computing Machinery, New York, NY, USA, 1089–1095.  https://doi.org/10.1145/3328778.3366927
  * Wohlin et al. (2022) Claes Wohlin, Marcos Kalinowski, Katia Romero Felizardo, and Emilia Mendes. 2022.  Successful combination of database search and snowballing for identification of primary studies in systematic literature studies.  _Information and Software Technology_ 147 (2022), 106908.
  * Yan et al. (2023) Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2023.  Practical and Ethical Challenges of Large Language Models in Education: A Systematic Literature Review.  arXiv:2303.13379 [cs.CL]
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022\.  Wordcraft: Story Writing With Large Language Models. In _27th International Conference on Intelligent User Interfaces_ (Helsinki, Finland) _(IUI ’22)_. Association for Computing Machinery, New York, NY, USA, 841–852.  https://doi.org/10.1145/3490099.3511105
  * Zan et al. (2023) Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023\.  Large language models meet NL2Code: A survey. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 7443–7464.
  * Zastudil et al. (2023) Cynthia Zastudil, Magdalena Rogalska, Christine Kapp, Jennifer Vaughn, and Stephen MacNeil. 2023\.  Generative AI in Computing Education: Perspectives of Students and Instructors.  _IEEE Frontiers in Education (FIE)_ (2023).  https://doi.org/10.48550/arXiv.2308.04309
  * Zhang et al. (2022) Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2022.  Repairing Bugs in Python Assignments Using Large Language Models.  _arXiv preprint arXiv:2209.14876_ (2022).

##  Appendix A Paper Extraction Form

The following questions were placed on a form that the group used when
evaluating the papers that were included in the literature review.

  1. (1)

Paper title [text entry]

  2. (2)

Bibtex entry [text entry]

  3. (3)

Article type [multi-select]

     * •

Position / discussion paper

     * •

Supervised study (a study that is conducted in a highly controlled environment
like a research lab)

     * •

Unsupervised study (a study that is conducted in a less restricted
environment, such as online)

     * •

New tool paper (presenting a new tool / LLM)

     * •

Evaluation paper (evaluating an existing tool / LLM)

     * •

Other [text entry]

  4. (4)

Author affiliation [multi-select]

     * •

Academic

     * •

Industry

  5. (5)

Country of human participants: If data is collected from human participants,
provide a comma separated list of countries of where participants were located
(if not explicitly mentioned, put “unclear”). [text entry]

  6. (6)

Level of human participants [multi-select]

     * •

Uncontextualized

     * •

Primary school (e.g. elementary, intermediate, middle school)

     * •

Secondary school (e.g. high school)

     * •

Tertiary education (e.g. college, university)

     * •

Informal education (e.g. MOOCs)

     * •

Professional developers

     * •

Not applicable

     * •

Other [text entry]

  7. (7)

Number of human participants from whom data was collected (if available, type
into other) [multi-select]

     * •

Not applicable

     * •

Unclear

     * •

Other [text entry]

  8. (8)

Description of participants: A copy-paste (or paraphrased) description of
participants from whom data is collected which may be useful to a more
detailed thematic analysis. This information can often be found at the
beginning of a Methods section. [text entry]

  9. (9)

How do the authors motivate the work: A copy-paste (or paraphrased)
description of the motivation for the work as expressed by the authors. [text
entry]

  10. (10)

What LLM / tool is used? [multi-select]

     * •

GPT-3

     * •

GPT-4

     * •

Codex

     * •

Copilot

     * •

Unclear

     * •

N/A

     * •

Other [text entry]

  11. (11)

What are the explicit research questions / research goals / hypotheses in the
article? A copy-paste (or paraphrased) description of the RQs, goals,
hypotheses. [text entry]

  12. (12)

What programming languages are involved in the study? [multi-select]

     * •

Java

     * •

Python

     * •

C

     * •

C++

     * •

Not programming language focused

     * •

Other [text entry]

  13. (13)

How does the article evaluate the data collected? [multi-select]

     * •

Qualitatively

     * •

Quantitatively

     * •

N/A

  14. (14)

Quality assessment: An assessment of the research “quality”. [Yes / No / Vague
radio grid]

     * •

Is there a clearly defined research question/hypothesis?

     * •

Is the research process clearly described?

     * •

Are the results presented with sufficient detail?

     * •

Are threats to validity / limitations addressed in an explicit (sub)section?
(code as “vague” if discussed, but not in a separate subsection)

  15. (15)

What is the contribution / what are the key results of the article? Provide a
short summary of the main findings. [text entry]

  16. (16)

Curriculum changes: Provide any potential effects on computing curriculum that
could result from this work (when answering this question, please feel free to
provide some of your own commentary - it is fine to mention curriculum changes
which the paper prompted you to think about, even if they aren’t explicitly
mentioned by the paper authors). [text entry]

  17. (17)

Opportunities: Provide any potential opportunities for computing education
that could result from this work (when answering this question, please feel
free to provide some of your own commentary - it is fine to mention
opportunities which the paper prompted you to think about, even if they aren’t
explicitly mentioned by the paper authors). [text entry]

  18. (18)

Threats: Provide any potential threats for computing education that could
result from this work (when answering this question, please feel free to
provide some of your own commentary - it is fine to mention threats which the
paper prompted you to think about, even if they aren’t explicitly mentioned by
the paper authors). [text entry]

  19. (19)

Additional notes: Can be used to note any interesting aspects of paper or
anything else relevant that isn’t captured in the extraction fields above.
[text entry]

##  Appendix B Student Survey Questions

The following questions were used in the survey filled out by students.

  1. (1)

Gender

     * ○

Man

     * ○

Woman

     * ○

Non-binary

     * ○

Other:

  2. (2)

Country (drop-down list)

  3. (3)

Level of Study

     * ○

First Year

     * ○

Second Year

     * ○

Third Year

     * ○

Fourth Year

     * ○

Fifth Year

     * ○

Other:

  4. (4)

Degree Major / Specialization

     * ○

Computer Science

     * ○

Software Engineering

     * ○

Information Technology

     * ○

Computer Engineering

     * ○

Bioinformatics

     * ○

Other:

  5. (5)

Number of courses with a programming component which you have completed (text
entry)

  6. (6)

Rate your agreement with the following statements: (Likert)

     * •

I regularly use GenAI tools when working with text (e.g.: writing emails,
reports, summaries)

     * •

I regularly use GenAI tools when working with code (e.g.: generating code or
explanations, writing programs, debugging, …)

     * •

I regularly use GenAI tools when working with images (e.g.: generating new
pictures, …)

  7. (7)

After generating code using GenAI tools, I mostly:

     * ○

Not applicable (I have not used GenAI tools to generate code)

     * ○

Use the code immediately.

     * ○

Skim through the code briefly to make sure that it looks correct.

     * ○

Read it carefully (with skepticism) to ensure that it is correct.

     * ○

Read it carefully (with skepticism) and also write code to test it.

  8. (8)

Rate your agreement with the following statements: (Likert)

     * •

I expect to use GenAI tools increasingly in my learning practices in the
future

     * •

Using GenAI tools frequently to generate code is harmful for my learning of
programming

     * •

GenAI tools can provide guidance for coursework as effectively as human
teachers

     * •

GenAI tools will replace human teachers in the future

  9. (9)

Students must be taught how to use GenAI tools well for their future careers
(Likert)

  10. (10)

When you have a question regarding the material you are studying or are stuck
on a problem, in what order do you do the following? (ranking question)

     * •

Ask using GenAI tools

     * •

Ask on the course discussion forum

     * •

Search online (e.g. Google)

     * •

Ask a friend

     * •

Ask on online forums like StackOverflow

     * •

Ask the course instructor/TA

  11. (11)

Rate your agreement with the following statements (Likert)

     * •

The policies at my university are clear regarding what is allowed and what is
not allowed in terms of using GenAI tools

     * •

The policies in the courses I took last semester were clear regarding what is
allowed and what is not allowed in terms of using GenAI tools

     * •

There should be no restrictions on the use of GenAI tools in coursework

  12. (12)

For programming assignments, I believe GenAI should be:

     * ○

Always allowed

     * ○

Allowed in some assignments, disallowed in others (based on the assignment
type, course level, etc.)

     * ○

Always disallowed

  13. (13)

Can you elaborate on when you believe GenAI should be allowed or disallowed?
(text entry)

  14. (14)

To what extent do you think students at your school are using GenAI tools in
ways that your instructors would not approve of?

     * ○

Almost everyone

     * ○

Many

     * ○

Some

     * ○

A few

     * ○

Almost none

  15. (15)

In the absence of an explicit course policy on the use of GenAI tools, which
of the following do you consider as NOT ethical? (Mark all that apply):

     * □

It is unethical to auto-generate a solution for the whole assignment (or a
large portion of it) and submitting it without understanding it.

     * □

It is unethical to auto-generate a solution for the whole assignment (or a
large portion of it) and submitting it after reading it and completely
understanding it.

     * □

It is unethical to auto-generate a solution even for small parts of the
assignment.

     * □

It is unethical to use GenAI tools to ”explain” to you (step-by-step) how to
solve the problem.

     * □

It is unethical to provide your code to GenAI tools and ask them to help you
fix a bug.

     * □

It is unethical to ask GenAI tools to comment, tidy and improve the style of
your code.

     * □

It is unethical to write the solution in a programming language (other than
the one used in the course) and asking GenAI tools to translate it for you to
the language of the course (and then submitting the translated code).

  16. (16)

If everyone in class is using GenAI tools, but it is against the rules to use
them, then I would still use them. (Likert)

  17. (17)

Rate your agreement with the following statements (Likert)

     * •

GenAI tools will negatively impact my future job prospects

     * •

GenAI tools will harm the development of generic or transferable skills such
as teamwork, problem-solving, and leadership

     * •

I am concerned that I will become over reliant on GenAI tools

     * •

I trust the code written by GenAI tools more than the code I write

     * •

My instructors can detect code that was written by GenAI tools

     * •

My instructors actively check for unauthorized use of GenAI tools

     * •

Even if my instructors disallow GenAI tools in my programming assignments, it
is fine for me to use them to generate code as long as I understand the code
very well. It is unethical only if I copy without understanding.

  18. (18)

Describe the ways you currently use GenAI tools in computing courses for text
generation (e.g.: writing reports, summaries, etc) (text entry)

  19. (19)

Describe the ways you currently use GenAI tools in computing courses for code
generation (e.g.: debugging, writing, etc) (text entry)

  20. (20)

Describe the effects you think GenAI tools will have on your prospects for
future employment: (text entry)

  21. (21)

What are your views on the allowed usage of GenAI tools in coursework/exams?
(text entry)

##  Appendix C Instructor Survey Questions

The following questions were used in the survey filled out by instructors.

  1. (1)

Gender

     * ○

Man

     * ○

Woman

     * ○

Non-binary

     * ○

Other:

  2. (2)

Country (drop-down list)

  3. (3)

Teaching Experience (years) (text entry)

  4. (4)

Select the sizes of classes you taught in the most recent semester

     * □

1-10 students

     * □

11-30 students

     * □

31-50 students

     * □

51-100 students

     * □

101-250 students

     * □

251-500 students

     * □

500+ students

  5. (5)

Select the type of department/school/faculty

     * ○

Computer Science

     * ○

Software Engineering

     * ○

Information Technology

     * ○

Computer Engineering

     * ○

Bioinformatics

     * ○

Other

  6. (6)

Rate your agreement with the following statements: (Likert)

     * •

I regularly use GenAI tools when working with text (e.g.: writing emails,
reports, summaries)

     * •

I regularly use GenAI tools when working with code (e.g.: generating code or
explanations, writing programs, debugging, …)

     * •

I regularly use GenAI tools when working with images (e.g.: generating new
pictures, …)

  7. (7)

After generating code using GenAI tools, I mostly:

     * ○

Not applicable (I have not used GenAI tools to generate code)

     * ○

Use the code immediately.

     * ○

Skim through the code briefly to make sure that it looks correct.

     * ○

Read it carefully (with scepticism) to ensure that it is correct.

     * ○

Read it carefully (with scepticism) and also write code to test it.

  8. (8)

Rate your agreement with the following statements: (Likert)

     * •

I expect to use GenAI tools increasingly in my teaching practices in the
future

     * •

Using GenAI tools frequently to generate code is harmful for my students’
learning of programming

     * •

GenAI tools can provide guidance for coursework as effectively as human
teachers

     * •

GenAI tools will replace human teachers in the future

  9. (9)

Rate your agreement with the following statements

     * •

Students must be taught how to use GenAI tools well for their future careers

     * •

I plan to change my assessment practices now that GenAI tools are commonly
available

     * •

I plan to change my curriculum now that GenAI tools are commonly available

  10. (10)

Rate your agreement with the following statements (Likert)

     * •

The policies at my university are clear regarding what is allowed and what is
not allowed in terms of using GenAI tools

     * •

The policies in the courses I taught last semester were clear regarding what
is allowed and what is not allowed in terms of using GenAI tools

     * •

There should be no restrictions on the use of GenAI tools in coursework

  11. (11)

For programming assignments, I believe GenAI should be:

     * ○

Always allowed

     * ○

Allowed in some assignments, disallowed in others (based on the assignment
type, course level, etc.)

     * ○

Always disallowed

  12. (12)

Can you elaborate on when you believe GenAI should be allowed or disallowed?
(text entry)

  13. (13)

To what extent do you think students at your school are using GenAI tools in
ways that you would not approve of?

     * ○

Almost everyone

     * ○

Many

     * ○

Some

     * ○

A few

     * ○

Almost none

  14. (14)

In the absence of an explicit course policy on the use of GenAI tools, which
of the following do you consider as NOT ethical for students to do? (Mark all
that apply):

     * □

It is unethical to auto-generate a solution for the whole assignment (or a
large portion of it) and submitting it without understanding it.

     * □

It is unethical to auto-generate a solution for the whole assignment (or a
large portion of it) and submitting it after reading it and completely
understanding it.

     * □

It is unethical to auto-generate a solution even for small parts of the
assignment.

     * □

It is unethical to use GenAI tools to ”explain” how to solve the problem step-
by-step.

     * □

It is unethical to provide code to GenAI tools and ask them to help fix a bug.

     * □

It is unethical to ask GenAI tools to comment, tidy and improve the style of
the code.

     * □

It is unethical to write the solution in a programming language (other than
the one used in the course) and asking GenAI tools to translate it to the
language of the course (and then submitting the translated code).

  15. (15)

Rate your agreement with the following statements (Likert)

     * •

GenAI tools will negatively impact my students’ future job prospects

     * •

GenAI tools will harm the development of generic or transferable skills such
as teamwork, problem-solving, and leadership

     * •

I am concerned that my students will become over reliant on GenAI tools

     * •

I trust the code written by GenAI tools more than the code I write

     * •

I can detect code that was written by GenAI tools

     * •

I actively check for unauthorized use of GenAI tools

  16. (16)

Describe the ways you currently use GenAI tools in computing courses for text
generation (e.g.: writing reports, summaries, etc) (text entry)

  17. (17)

Describe the ways you currently use GenAI tools in computing courses for code
generation (e.g.: debugging, writing, etc) (text entry)

  18. (18)

Please describe any changes you have made, or plan to make, to your teaching
approaches in courses you are teaching: (text entry)

  19. (19)

Please describe any changes you have made, or plan to make, to your assessment
approaches in courses you are teaching: (text entry)

  20. (20)

If you have already implemented changes, describe how successful you think
they were. (text entry)

  21. (21)

What new content/courses do you think should be taught/added to the
curriculum? (text entry)

  22. (22)

Does your institutional academic integrity policy explicitly mention
generative AI?

     * ○

Yes

     * ○

No

  23. (23)

Can you provide a link to your institution’s policy on GenAI tools? (text
entry)

  24. (24)

Do you have a policy explicitly stated in your syllabus about when GenAI tools
should or could be used in your course?

     * ○

Yes

     * ○

No

  25. (25)

If you have a statement about acceptable use of generative AI in your course
syllabus documents, and you are willing to share that statement, please paste
it below: (text entry)

  26. (26)

Have you observed any students using these models? If so, how are they using
them? (text entry)

  27. (27)

Are you willing to be interviewed regarding the use of GenAI tools in
computing classes? If so, please provide a preferred contact email address.
(text entry)

##  Appendix D Student Guide

Generative AI refers to a kind of artificial intelligence software that is
capable of generating information in response to prompts. The software is
trained on source data, and uses that training data as input to a
sophisticated model that predicts the appropriate response to the prompt. It
does not understand the prompts, but it produces a convincing simulation of
understanding. Examples of generative AI systems that use text include ChatGPT
and Bard, and generative AI models capable of generating images include
Midjourney and DALL-E.

Generative AI tools can be used in ways that increase productivity and help
you to learn. However, they may also be used in unproductive ways that provide
answers without helping you to learn.

Policy on generative AI: • You may use AI tools to help you learn during lab
exercises and assignments. • You will NOT be permitted to use AI tools in
secure assessments (i.e., the Test and Exam).

### Examples of productive use

Generative AI tools are used in industry so you will be likely to use them
regularly in your future work after graduation. Therefore, you should learn to
use them appropriately so you receive the most long-term benefit. As a
student, effective uses of generative AI tools are centered on helping you
understand course material, and may include asking generative AI to:

  * •

Explain a given topic, or to provide an example of how programming constructs
are used.

  * •

Explain your program one line at a time.

  * •

Produce an example that is similar to assignment questions.

  * •

Explain the meaning of error messages.

  * •

Generate code to complete tasks that you have already mastered from previous
coursework.

### Examples of inappropriate use

Some uses of generative AI do not typically help you learn, and such uses are
likely to result in worse long-term outcomes (e.g., you will not be able to
complete Test and Exam questions, or to continue to following courses that
expect a mastery of early programming content). Examples of these uses are:

  * •

Using AI tools on official assessments where it has been forbidden.

  * •

Asking generative AI to complete laboratory questions or assignments for you.

  * •

Asking generative AI to debug code that has errors.

  * •

Writing a code solution in a language you know and then asking an AI tool to
translate that code into the language required for the assignment.

### Risks of generative AI

There are many risks associated with the use of generative AI.

Accuracy:

If you are using generative AI tools for learning then you should always
double-check the content. For example, if you are assigned to write a program
that uses a specific algorithm, AI tools may generate a solution that arrives
at the correct answer but does not use the required algorithm. If you use
generative AI to assist in the creation of assessed content then you are
responsible for the accuracy and correctness of the work that you submit.

Quality:

Content generated may be of poor quality, and generic in nature. Code may have
security flaws and may contain bugs. It is important that you understand how
any generated code works and you evaluate the quality of the content.

Learning:

Generative AI can be a powerful productivity tool for users who are already
familiar with the topic of the generated content because they can evaluate and
revise the content as appropriate. Tasks assigned by your teachers are
designed to help you learn, and relying on AI tools to complete tasks denies
you the opportunity to learn, and to receive accurate feedback on your
learning.

Over-reliance:

Using AI tools to do your work for you may achieve the short-term goal of
assignment completion, but consistent over-reliance on AI tools may prevent
you from being prepared for later examinations, subsequent coursework, or
future job opportunities.

Motivation:

You may experience lack of motivation for tasks that generative AI can
complete. It is important to understand that you need to master simple tasks
(which generative AI can complete) before you can solve more complex problems
(which generative AI cannot complete). Stay motivated!

### Impact on others

There are many consequences to inappropriate usage of AI tools. Some of these
consequences may be unintended, and could potentially harm others. For
example:

Other students:

You could expose other students to harm by preventing their learning or
including content in a group assignment that violates academic integrity.

Faculty:

Violating academic integrity standards through the use of AI tools requires
time and energy, and is emotionally draining to teachers and administrators,
to enforce these standards.

Institutional:

Including code from AI tools that you do not understand could expose the
university to loss of reputation or even financial harm through lawsuits.

### Academic misconduct

Using generative AI in ways that are not permitted will be treated as academic
misconduct. This will have serious consequences.