Harry Ye

Research

Research

My research interests are Human-Computer Interaction (HCI), with a current focus on education technology, and Human-Centred AI.

Current Projects

Design, Implementation, and Evaluation of a Novice Systematic Review Assistant

Advised by Prof. Tingting Zhu, Carolina Nobre, and Michael Liut

Papers

A screenshot of an academic research tool called "ARC: Automated Review Companion." The interface is divided into several sections, including the Search Bar, Search History, Search Results, and a section for Paper Filter Questions (LLM-Powered). Search Bar: This section allows users to input an advanced search query. The query shown reads: AI AND "Machine Learning" AND NOT education AND NOT chatbot. There are filters for database types (e.g., DBLP, Semantic Scholar, Web of Science, IEEE Xplore, Scopus) and a field for entering validation papers using DOI numbers (such as 10.1109/ACCESS.2021.3053725). Users can set the publication year range (in this case, 2024) and conduct a search or clear the form. Buttons labeled Search and Clear are located at the bottom right. Search History: This area shows a history of search queries conducted, such as "Search 1" and "Search 2", each labeled with a reference code and a description of the search terms used. Search Results: There are 999 publications found based on the search. The results are shown in a table format, with each row displaying a paper ID, title, and source. For example, the first result is titled "Evaluation of Convolutional Neural Network Model and Various Machine Learning Models for Decision-Making in the Health Care Industry" from the Semantic Scholar database. Buttons to Select All, Forward Search, Backward Search, and options to show or edit metadata are available. There is also an export function, allowing users to download results in CSV, BibTeX, or RIS formats. Paper Filter Questions (LLM-Powered): A section where users can apply filters to the search results using questions powered by a language model. One question asks, "Is the paper or experiment related to postsecondary study?" with options to answer "Yes," "No," or "Uncertain." The table shows the Paper ID, Paper Title, and corresponding answers with rationale. For example, one paper titled "Artificial Intelligence and the Future of Work" has an answer of "Uncertain" with a note that the abstract does not provide enough information. At the bottom of this section, there is a Submit Questions button for users to finalize their answers.

ARC: an Automated Review Companion Leveraging User-Centered Design for Research Integration (In Submission)

Ye, R., Sibia, N., Zavaleta Bernuy, A., Zhu, T., Nobre, C., & Liut, M. (2024). OSF Preprint.

Preprint
ARC Deployment

Project Description

We developed ARC, an open-source Automated Review Companion designed to streamline and enhance the process of conducting systematic literature reviews (SLRs), particularly in the computing field. SLRs are essential for synthesizing research, but they can be time-consuming and labor-intensive. ARC addresses these challenges by automating tasks such as literature searches, data extraction, and reference tracking, making the process more efficient while maintaining high standards of transparency and reproducibility.

Developed using a user-centered design approach, ARC was iteratively refined through feedback from 20 experienced computing researchers. Key features of ARC include support for multiple scholarly databases, automated forward and backward reference searching, and metadata extraction. The tool incorporates FAIR (Findable, Accessible, Interoperable, and Reusable) principles, ensuring that it promotes open, reproducible research practices. Overall, ARC offers a meaningful improvement in the usability and effectiveness of tools for systematic reviews, lowering the barrier to conducting thorough, rigorous research in computing and other fields.

Past Projects

Design and Evaluation of New Programming Tools for Novices using AI Coding Assistants

Advised by Prof. Tovi Grossman, Michelle Craig

Mentored by Ph.D. student Majeed Kazemitabaar

Papers

The image is a screenshot of the CodeAid tool interface, showing a programming environment with two distinct panels. On the left panel is a segment of code titled "Code to ask about:" which contains a C function 'fib' that calculates Fibonacci numbers and assigns them into a dynamically allocated array on the heap. The code includes pointers, memory allocation, and a loop structure. The right panel has a "Question from code:" box with the user's query about what '*heap_pt' is pointing to within the array. Below this, there are several interactive buttons for different types of assistance, including asking a question, asking a question from code, explaining code, helping fix code, and helping write code. The interface is designed with a dark theme and the text and interactive elements are highlighted with blue and teal accents for clarity and visual appeal.

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs.

Kazemitabaar, M., Ye, R., Wang, X., Henley, A., Denny, P., Craig, M., & Grossman, T. (2024, May). In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24)

ACM DL
Preprint
Dataset

Project Description

We introduced CodeAid — An LLM-based programming assistant. CodeAid was deployed in a large university classroom of 700 students and conducted a thematic analysis of its 8,000 usages, supplemented by weekly surveys and student interviews. Further feedback was obtained from eight programming educators. Results showed that most students used CodeAid for understanding concepts and debugging, though some students directly asked for code solutions. While educators valued its educational merits, they also highlighted concerns about occasional inaccuracies and the potential for students to rely on tools like ChatGPT.

Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons

Advised by Prof. Joseph Jay Williams, Andrew Petersen, Michael Liut

Mentored by Ph.D. Candidate Angela Zavaleta-Bernuy

Papers

first page of presentation for paper titled: student Interaction with Instructor Emails in Introductory and Upper-Year computing courses at SIGCSE 2024
SIGCSE 2024 figure: Comparison of two reminder emails. Two versions of the email are located on the left and right of the figure, with the middle section highlighting the similarities and differences between the two emails. Left Email (Control Group): Subject: [DB course code] Upcoming Deadlines Content: 'This week in DB the following assessments are due: [Thursday, Feb 17th, 12:00pm] Prep Week 5; [Friday, Feb 11th, 12:00pm] Lab 2.' Reminder to complete readings from Chapter 5 (5.1-5.6) and try some problems from Ex. 5.1-5.3. Reference to programming with 'AI Copilot on GitHub' with suggestions for code lines or functions inside the editor. Sign-off: 'All the best, [DB] teaching team.' Postscript praising the reader's progress, a reminder that the email is sent to all students, a contact email (redacted), and an unsubscribe link. Right Email (Treatment Group): Subject: [DB course code] Programming with AI and Upcoming Deadlines Content: Begins with 'Good Evening!' Introduces 'AI Copilot on GitHub' emphasizing its newness, its usefulness to experienced programmers, and those unfamiliar with syntax. Lists the same assessments due: [Thursday, Feb 17th, 12:00pm] Prep Week 5; [Friday, Feb 11th, 12:00pm] Lab 2. Reiterates the reminder about Chapter 5 readings (5.1-5.6) and exercises. Adds a note about visiting office hours and a motivational statement: 'One month down, only two more to go! You've got this! happy face' Sign-off: 'All the best, [Instructor's name].' Shares a quote: '“Before software should be reusable, it should be usable.” - Ralph Johnson.' Ends with the same postscript as the control email, praising progress, informing the email goes to all, a redacted contact email, and an unsubscribe link. Blue Boxes compare the two versions of email in the middle: More descriptive subject line: The treatment email's subject includes 'Programming with AI.' Extra greeting: The treatment email starts with 'Good Evening!' Same resources, with a more personal touch: The treatment email explains 'AI Copilot on GitHub' in a more personalized manner. Same course reminder: Both emails list the assessments due. Extra instructor reminder: The treatment email suggests visiting office hours. More personalized sign-off and extra quote: The treatment email includes an instructor's name in the sign-off and a quote by Ralph Johnson. Same disclaimer and unsubscribe link: Both emails provide an unsubscribe option.

Student Interaction with Instructor Emails in Introductory and Upper-Year Computing Courses.

Zavaleta Bernuy, A., Ye, R., Sibia, N., Nalluri, R., Williams, J. J., Petersen, A., Smith, E., Simion, B., & Liut, M. (2024, March). In Proceedings of the 55th ACM Technical Symposium on Computer Science Education (SIGCSE ’24).

Paper
Slides
first page of presentation for paper titled: Do Students Read Instructor Email? A Case Study of Intervention Email Open Rates
This image is a graphical representation of a "Data Collection Timeline" for an academic study. The timeline spans from Week 1 to Week 12 of a term. It's divided into two sections: two-thirds of participants received emails (treatment group), while one-third did not receive any emails (control group). The intervention with emails started in Week 5, coinciding with the course becoming more challenging. This is noted as a time when instructors typically see students start to struggle. Key elements and milestones are marked as follows: Start of Term: Week 1, with a START sign. The sending of Email #1 in Week 5, with a dashed line indicating the action post-Week 5. Email #2 is sent in Week 7, again indicated with a dashed line post-Week 7. Email #3 is sent in Week 10, indicated with a dashed line pre-Week 10. End of Term: Week 12, marked with a finish line labeled "THE END." Additionally, there are specific measurements highlighted in blue boxes: Email open rates: tracking whether a student opened the email. Link click rates: tracking whether a student clicked on at least one link in the email. At the end of the term, two evaluations are noted: A student survey regarding email usage and course intentions. Instructors' perspectives, where they predict student email engagement. The accompanying notes indicate that no emails were sent during Weeks 6 and 9 due to term tests and none during Week 8 due to a university-wide study break. Pre-lecture work is assigned on Mondays, and post-lecture work is due on Fridays, as indicated at the top of the timeline.

Do Students Read Instructor Emails? A Case Study of Intervention Email Open Rates

Zavaleta Bernuy, A., Ye, R., Tran, E., Mandal, A., Shaikh, H., Simion, B., Petersen, A., Liut, M., & Williams, J. J. (2023, November). In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (Koli Calling ’23).

Paper
Slides
first slide for paper titled: Behavior consequences of Reminder EMail on Students' Academic Performance: a Real-world deployment

🏆 Behavioral Consequences of Reminder Emails on Students’ Academic Performance: a Real-world Deployment

Ye, R., Chen, P., Mao, Y., Wang-Lin, A., Shaikh, H., Zavaleta Bernuy, A., & Williams, J. J. (2022, September). In Proceedings of the 23rd Annual Conference on Information Technology Education (SIGITE ’22).

Paper
Slides

Posters

Chen, P.*, Ye, R.*, & Mao, Y.* (2021 March). Behavioral consequences of reminder emails on students’ performances for homework: a real-world deployment. University of Toronto Undergraduate Poster Fair.

*contributed equally

ROP-Poster-March-2021Download

Project Description

The Intelligent Adaptive Interventions (IAI) Lab’s OnTrack project aims to support and improve students’ learning behaviours through homework reminder emails. 

Randomized A/B experiments provide one source of evidence for designing instructional interventions and understanding how students learn. Generally, conducting experiments in the field raises the question of how the data can be used to rapidly benefit both current and future students. Adaptive experiments provide the solution as these can be deployed to increase the chances that current students also obtain better learning outcomes. Our goal is to provide a standalone system that can send students reminders, and prompts to reflect. We know instructors have their own style and approach to messaging, and so this OnTrack messaging service is meant to be a separate channel that is clearly different from the high-priority announcements from instructors. 

OnTrack utilizes randomized A/B comparisons to evaluate different reminders or alternative ideas about how best to encourage students to start work earlier and, crucially, to measure the impact of these interventions on student behaviour. The effect of homework email reminders on students is further evaluated by conducting an adaptive experiment using multi-armed bandit algorithms which identifies and allocates more students to the most optimal treatment. The rationales are:

  1. Students can receive extra support with everything being online, with the IAI lab putting energy into crafting different messages, based on student feedback.
  2. OnTrack emails won’t ‘interfere’ with instructor communication or make students start to ignore them (concerns raised by instructors, which we share).
  3. Instructors still have the option to get data from what we test out and give suggestions or requests for what emails they want to see us test out. 

After several deployments at UTSG and UTM, OnTrack has gathered valuable understandings of how students perceive different aspects of reminder emails, i.e. subject lines, calendar events, links to online homework, motivational messages, etc. Currently, OnTrack is pioneering and experimenting with additional ideas, such as incorporating cognitive psychology aspects in the design of the emails.