We are aiming to investigate how prompting participants to reflect on their daily AI usage goals and providing visual feedback (via hand-drawn brains) based on their self-assessed performance influences both AI usage trends and self-awareness of AI productivity over 7 days.
Research Question:
Does a reflective intervention with visual feedback improve how effectively participants use AI tools to achieve their goals?
The process
In order to get to this study, we freelisted interventions with sticky-notes. Then, we chose our four strongest ideas to storyboard: GPT study-mode scaffolding, a tutor for your prompts, a brain rot indicator, and a self-reflection quiz. In the storyboards, the red marker indicates the moment that we intervene. This allowed us to really empathize with our users: to get in their head, imagine their narrative. With our storyboards, we generated pros and cons for each solution.
For our final idea, we decided to combine two solutions: a Brain Rot indicator on the users’ devices that visually displays how healthy their brain is as they progressively interact with AI tools, and a Self-Reflection quiz at the beginning and the end of the day that allows users to reflect on their intentions before interacting with AI models that day, and reflect on whether they achieved these intentions at the end of the day.
The reason why we chose these two particular solutions was because we did not want our solution to unintentionally manipulate the user. By allowing users to set their own intentions, we ensure that we are not imposing any particular way of using AI. Rather, our intervention is about helping the user meet their own expectations of themselves. Furthermore, by giving a visual, visceral representation of the success of the goal, we can harness the “glimmer” effect – that hit of dopamine that allows habit-formation. As we found from our baseline synthesis, more effective AI usage can be a low-motivation but high ability habit – therefore, our goal for our intervention is to generate more motivation for behavior change.
Figure 1. Custom GPT scaffolding storyboard.
This solution is similar to ChatGPT’s study mode: it would be a wrapper that never gives students the direct answers, should they ask for them. Since this would block students from using AI poorly, this would help students be more mindful of their AI usage. However, this solution is easy to game, relying too much on the user’s goodwill. If they want to find the answer, for example, they could simply go to another website. Therefore, this solution would only work for already highly-motivated students. In other words, in the B=MAT equation, this would only give students the ability to interact with AI in a different way – but this would not improve their motivation. As mentioned above, effective AI usage is a high-ability (most people already know what’s better) but low-motivation behavior change.
Figure 2. Tutor your prompts! storyboard.
This solution would be an extension on pre-existing websites that monitor your prompts to ChatGPT or other AI tools. However, there were many problems with this solution, the first being that this would be way too annoying for the user in real life. Furthermore, this solution is also too prescriptive: we would have to continually suggest new ways of controlling the user’s behavior, which might not work for every user. This solution also seems like a bandaid for the real issue, by using AI to solve AI.
Figure 3. Brain rot indicator.
This solution would be a visual indicator of how well the user has used AI throughout the day. Some pros of this solution is that it is visual, and therefore visceral. However, this solution has the same problem as the above solution: it is extremely prescriptive. We would have to dictate what “good” or “effective” AI usage means for the user. Therefore, we combined it with the self-reflection quiz, the only solution that allows the user to set their own goals. Another con is that users might not buy into the UI brain, making the happiness/unhappiness of the UI brain unaffecting. To combat this, personalizing the UI brain might make it easier for the user to get attached.
Figure 4. Self reflection quiz
In this solution, the user sets their own intentions for how they would use AI. This solution is most similar to other mindfulness solutions, which introduce friction into “easy” habits, like checking Instagram. Through the quiz, users would set their own goals for what they want to use AI for; by doing the quiz, users become more mindful of how they use AI and what learning, AI, etc. means to them. The con of this solution is that users might change their mind; they might skip through the quiz; but the pro of this solution is that no matter what, their behavior will be changed in some way: the added friction will help users self-reflect.
7-Day Intervention Study Design
| Day | Morning Prompt | Evening Prompt | Visual Feedback |
| 1 | “In one sentence, how do you intend to use AI today?” | “How successful were you in accomplishing your AI use goal today? (1 – terrible, 2 – poorly, 3 – OK, 4 – good, 5 – really good!)” | Brain image based on rating: 1 → rotted brain, 5 → strong/happy brain |
| 2 | Repeat morning prompt | Repeat evening prompt | Updated brain image |
| 3 | Repeat morning prompt | Repeat evening prompt | Updated brain image |
| 4 | Repeat morning prompt | Repeat evening prompt | Updated brain image |
| 5 | Repeat morning prompt | Repeat evening prompt | Updated brain image |
| 6 | Repeat morning prompt | Repeat evening prompt | Updated brain image |
| 7 | Repeat morning prompt | Repeat evening prompt | Updated brain image + cumulative brain (de)growth over the week |
Data Collection Plan
Data Collected Each Day:
- Morning: Goal statement (text)
- Recurring text 9 AM: “In one sentence, how do you intend to use AI today? (example: get step by step help on my 224N p-set today)
- Evening: Self-rating (1–5) and reflection
- Recurring text 9 PM: “How successful were you in accomplishing your AI use goal today? (1 – terrible, 2 – poorly, 3 – OK, 4 – well, 5 – really well!)
- Reflection on AI usage (short text: what worked, what didn’t): “Please briefly describe what worked and what didn’t.
- Optional: Log AI usage data (ask participants to send their AI chat logs via PDF)
Analysis Focus:
- Daily rating trends: we want to know how people’s intentionality changes over time
- Relationship between goal specificity and effectiveness rating: we want to know how intentional goal-setting impacts effectiveness – do more specific, intentional goals produce better outcomes?
- Impact of visual feedback (images of brain rotting) on self-assessment and AI engagement: we want to know whether the visual feedback gives users the “glimmer,” the motivation, to change their AI usage
Logistics:
- Team members will send daily text messages to their participants and add all of the responses into a Google Sheet: Intervention study data
Here are the introduction and closing messages:
Introduction Message
Dear Participant,
Thank you for expressing interest in taking part in our intervention study on AI use in academic work!
Please refer to the introduction document for more information about the purpose of the study, how to participate, expectations for the week, and how to get started. This study spans from Monday, February 9, through Sunday, February 15. At the beginning of each day, your moderator will reach out asking about your plans for AI usage for that day. At the end of each day, your moderator will ask you for your holistic evaluation on how successfully you felt you leveraged AI to accomplish your goals. Please respond with as much detail as possible.
If you have any further questions or concerns, please feel free to reach out to the study coordinator via text. Thank you once again, we greatly appreciate your insights.
Introduction document: Introduction Document: AI Intention and Reflection Study
Best,
Team Badger (CS 247B)
Closing Text
Dear Participant,
Thank you so much for taking the time to participate in our intervention study! We truly appreciate your willingness to share your experiences and insights with us.
We’ll be in touch with our findings soon! If you have any follow-up questions or would like to learn more about the study, please don’t hesitate to reach out.
Thank you again for your contribution. We couldn’t have done it without you.
Best,
Team Badger (CS 247B)
