Research informing the design of production AI systems grounded in structured reasoning and human-centered interaction, with publications at ICSE and ITiCSE.
Online educational resources (e.g., curricula, tutorials, documentation, Q&A sites) increasingly serve as key sources for secondary school students learning Computer Science Principles (CSP). A big obstacle to using these resources is finding information appropriate for the learning task and learner’s background. Research shows that secondary school students need support in searching the web and developing “information literacy” (e.g., find, evaluate, organize, use, and convey information). Exploratory search is particularly challenging as it goes beyond simple lookup and instead is comprised of learning and investigative search intents. A potentially promising approach uses a conversational search agent to conduct exploratory search through conversation with the learner. Research indicates that conversational agents can assist secondary school Computer Science (CS) teachers, especially those who lack CS background. In addition, research shows that secondary school students have high levels of engagement with conversational agents. Conversational agents could help alleviate social anxiety for students who do not want to ask teachers questions publicly. If designed with customization toward building better connections to learning, the agent could help students relate to the CS concepts to reflect on their own experiences as they relate to the CS concepts. The focus of this dissertation is Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search. The dissertation’s contributions include: (1) the design of a Web-based generative conversational search agent tailored to the CS domain using SE word embeddings, (2) an exploratory study with 18 CS stu- dents investigates the agent’s potential in supporting CS-related exploratory search, (3) a pedagogical fixed-response conversational search agent is designed for the CSP domain, aligning with Kaddoura’s Think-Pair-Share collaborative learning strategy, (4) metrics for evaluating conversational agent effectiveness and engagement, (5) a comparative study involving 45 secondary school students in a CSP class explores the use of conversational agents and web search, (6) a knowledge-based pedagogical generative conversational search agent for the CSP domain, utilizing retrieval augmented generation and prompt engineering, (7) an exploratory study with 20 CSP students examining the cus- tomization’s impact on aiding students in learning CSP concepts, and (8) an experience report of successes and challenges for future conversational agent researchers. The results of this dissertation show that CS students do find generative conversa- tional agents useful at helping with exploratory search, yet CS students believe that the current generation of Web-based generative conversational search agents are not effective in helping with exploratory search. Evaluation of our approach using effectiveness and engagement metrics indicate that generative conversational agents are highly effective and interactive, and are preferred over pedagogical fixed-response conversational search agents, yet generative conversational agents pose risks as learning tools. Students are dissatisfied with the interaction and effectiveness of pedagogical fixed-response conversational search agents; however, find them helpful at finding useful information. Further, results indicate that students preferred a customized knowledge-based pedagogical generative conversational search agent, with its terminology more suitable to secondary school level, examples more understandable, and better connections to personal experiences com- pared to a standard generative conversational agent.
@phdthesis{frazier2024investigating,title={Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search},author={Frazier, Matthew},year={2024},school={University of Delaware},publisher={University of Delaware},url={https://udspace.udel.edu/items/1d3815f9-1e86-4665-9ac1-18e81419da45},}
This paper explores leveraging conversational agents, specifically ChatGPT, to enhance the introduction of computing, focused on the Advanced Placement Computer Science Principles (CSP) course in secondary schools. Despite the potential benefits for diverse student audiences, little research has investigated their effectiveness and engagement in this context. We examine the customization of ChatGPT for secondary school CSP students, assessing its impact on exploratory searches for learning CSP concepts. Results from 20 high school students in grades 10-12 (ages 15-18) in a CSP course indicate that students preferred a customized ChatGPT, with its terminology more suitable to secondary school level, examples more understandable, and better connections to personal experiences compared to standard ChatGPT.
@inproceedings{10.1145/3649217.3653570,author={Frazier, Matthew and Damevski, Kostadin and Pollock, Lori},title={Customizing ChatGPT to Help Computer Science Principles Students Learn Through Conversation},year={2024},isbn={9798400706004},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3649217.3653570},doi={10.1145/3649217.3653570},booktitle={Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1},pages={633-639},numpages={7},keywords={chatgpt, computer science principles, conversational agent, exploratory search},location={Milan, Italy},series={ITiCSE 2024},}
Conversational agents that respond to user information requests through a natural conversation have the potential to revolutionize how we acquire new information on the Web (i.e., perform exploratory Web searches). Recent advances to conversational search agents use popular Web search engines as a back-end and sophisticated AI algorithms to maintain context, automatically generate search queries, and summarize results into utterances. While showing impressive results on general topics, the potential of this technology for software engineering is unclear.In this paper, we study the potential of conversational search agents to aid software developers as they acquire new knowledge. We also obtain user perceptions of how far the most recent generation of such systems (e.g., Facebook’s BlenderBot2) has come in its ability to serve software developers. Our study indicates that users find conversational agents helpful in gaining useful information for software-related exploratory search; however, their perceptions also indicate a large gap between expectations and current state of the art tools, especially in providing high-quality information. Participant responses provide directions for future work.
@inproceedings{10.1145/3510455.3512778,author={Frazier, Matthew and Kumar, Shaayal and Damevski, Kostadin and Pollock, Lori},title={Investigating User Perceptions of Conversational Agents for Software-Related Exploratory Web Search},year={2022},isbn={9781450392242},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3510455.3512778},doi={10.1145/3510455.3512778},booktitle={Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results},pages={51-55},numpages={5},keywords={wizard of Oz study, exploratory search, conversational search agent},location={Pittsburgh, Pennsylvania},series={ICSE-NIER '22},}
According to constructivist theory, students learn software security more effectively when examples are grounded in their own code. Generic examples often fail to connect with students prior work, limiting engagement and understanding. Advances in LLMs are now making it possible to automatically generate personalized examples by embedding security vulnerabilities directly into student-authored code. This paper introduces a method that uses LLMs to inject instances of specific Common Weakness Enumerations (CWEs) into students own assignment code, creating individualized instructional materials. We present an agentic AI framework, using autonomous LLM-based agents equipped with task-specific tools to orchestrate injection, evaluation, ranking, and learning outcome generation. We report the experience of deploying this system in two undergraduate computer science courses (N=71), where students reviewed code samples containing LLM-injected vulnerabilities and completed a post-project survey. We compared responses with a baseline using a widely adopted set of generic security instructional materials. Students qualitatively reported finding CWE injections into their own code more relevant, clearer, and more engaging than the textbook-style examples. However, our quantitative findings revealed limited statistically significant differences, suggesting that while students valued the personalization, further studies and refinement of the approach are needed to establish stronger empirical support.
@misc{frazier2026personalizingsecureprogrammingeducation,title={Towards Personalizing Secure Programming Education with LLM-Injected Vulnerabilities},author={Frazier, Matthew and Damevski, Kostadin},year={2026},eprint={2604.13955},archiveprefix={arXiv},primaryclass={cs.CR},}
Secondary school students enrolled in the AP Computer Science Principles (CSP) course commonly utilize web resources (e.g., tutorials, Q&A sites) to better understand key concepts in the curriculum. The primary obstacle to using these resources is finding information appropriate for the learning task and student’s background. In addition to web search, conversational agents are increasingly a viable alternative for CSP students. In this paper, we study the potential of conversational agents to aid secondary school students as they acquire knowledge on CSP concepts. We explore general purpose, generative conversational agents (e.g., ChatGPT) and custom, fixed-response conversational agents built specifically to aid CSP students. We present results from classroom use by 45 high school students in grades 9-11 (ages 14-17) across six CSP sections. Our main contributions are in better understanding how conversational agents can help CSP students and an evaluation of the effectiveness and engagement of different approaches for CSP exploratory search.
title = {Investigating Conversational Agents to Support Secondary School Students Learning CSP},author = {Frazier, Matthew and Damevski, Kostadin and Pollock, Lori},year = {2026},eprint = {2604.16213},archiveprefix = {arXiv},primaryclass = {cs.HC},}
This work informs my approach to designing production AI systems grounded in structured reasoning and human-centered interaction.
