Team Rhebok – Post-Study Interview Script

Link: Post-Study Interview Script

 

Post-Study Interview Script

 

Purpose

This interview explores how participants experienced the diary study itself and what it revealed about their habitual LLM use, especially moments before using an LLM, the presence or absence of independent thinking, and the emotional and cognitive experience of being asked to pause and reflect.

 

Questions

1. Overall experience of the diary study

Q1. Looking back on the five days as a whole, how was your diary experience?

• Did it feel reflective, intrusive, neutral, stressful, or insightful?
• Did your feelings about the study change over the five days?

 

2. Whether or not filling out the diary affected participant behavior

Q2. Each time you used an LLM, the diary asked you to pause and log your task and confidence beforehand. How did that pause affect you, if at all?

• Did it feel like a meaningful interruption or just a procedural step?
• Did it ever make you hesitate, reconsider, or reframe the task?

If needed:

• In moments where it did interrupt you, what changed in your thinking?
• In moments where it didn’t, why do you think the pause failed to register?

(We can directly examine whether the diary disrupted the habitual “LLM-first” loop.)

 

3. Whether or not there were specific tasks this affected MORE or LESS, specifically school vs personal life vs work

Q3 (Depends on Q2’s answer, will need to be framed as such): Were there any specific tasks where you feel that the diary study affected your LLM usage?

• This can be from any perspective, whether this is a reflective / confidence / stress / burdensome standpoint
• Were there tasks that you felt were completely unaffected? If so why?

 

4. If logging chatbot sessions revealed anything new to the user about their behavior or usage

Q4. Were there any additional trends or patterns that came about when filling out the logging form?

• Were there answers that were constant across the form?
• Were there answers that were extremely variant across the form?
• Did you report more LLM usage / fill out the form more on certain days?

 

Q5. Did the act of logging every LLM session reveal anything new about your own behavior or patterns?

• Frequency or timing of use
• Certain task types or situations
• How often have you used LLMs even when stakes were low

If needed:

• Was anything surprising or uncomfortable to notice?

 

6. Whether or not the study affected user behavior prior to using the LLM

Q6. During the study, what was your thought process right BEFORE you opened a chatbot?

• Did you generate an initial answer, plan, or hypothesis?
• Did you try to hinder yourself from using a chatbot?
• Or did you mostly rely on the LLM to start the thinking process?

If needed:

• When you didn’t think first, what made that feel reasonable or necessary in the moment?

(Related to our behavior change goal, we can locate where thinking begins.)

 

7. Whether or not the study led participants to change their behavior

Q7. Were there moments during the study when you paused and thought more deeply on your own before using the LLM?

• What triggered those moments?
• What did “thinking first” look like in practice?

Q8 (IF THEY SAY NO TO Q7). What would make that “pause and critical thinking” hard to execute?
Examples may include:

• Discomfort with ambiguity
• Perfectionism (needing high-quality immediately)
• Mental fatigue / low bandwidth
• Time efficiency
• Reward structure (LLM gives fast certainty)

(We can identify conditions that enable or block the desired habit.)

Q9. What would make pausing before using the chatbot more worth it/seem more valuable
Critical for habit formation: ☆☆ (Important for Intervention)

• Learning / skill growth
• A sense of authorship
• More trust in your own judgment
• Better long-term outcomes
• Avoiding errors or embarrassment
• A feeling of autonomy

 

10. Interpreting Confidence

Q10. When you rated your confidence before starting a task, how did you decide on that number?

• What did “confidence” mean to you: being correct, being efficient, producing something acceptable?

If needed:

• Looking back now, do you think those ratings reflected your actual capability, or more your comfort level at the time?

(We can unpack confidence as a multi-dimensional driver of LLM reliance.)

 

Q11. In sessions where you rated correctness confidence high, how did you decide it was correct?

• Did you check against sources?
• Did it match your prior beliefs?
• Did you rely on the model’s tone?
• Did you not evaluate at all?

 

Q12. Your ownership ratings (to what extent you think the output represents your own work), what makes something still feel like “mine”?

• Is it about generating the core idea yourself?
• Doing the first draft?
• Editing and reasoning?
• Understanding why it’s correct?

 

13. Emotional Experience of Reliance

Q13. Emotionally, what role did LLMs play for you during the study?

• Did they provide relief, reassurance, momentum, or reduced anxiety?
• Did those feelings ever make it easier to skip thinking things through on your own?

If needed:

• Were there moments where reliance felt uncomfortable or conflicted?

(We can investigate affective drivers of habitual use.)

14. How did the study change user’s perception of their reliance or reliance on chatbot uses?

Q14. After doing this diary week, do you feel any desire to change how you use LLMs, specifically to think independently first? Why?

• What feels compelling about that change?
• What feels unnecessary or unrealistic?

Q15. After completing this diary study, has your understanding of “over-reliance on LLMs” changed at all?

• Over-reliance as “I outsource judgment”
• “I don’t form my own view”
• “I accept answers without internal reasoning”

If needed:

• If so, how would you define over-reliance now, especially in terms of thinking?

(We can capture post-study conceptual shift.)

16. Thinking about how habits can be built

Q16. If we wanted to help you build a “think-first” habit, where in your workflow should the intervention live?
Examples may include:

• Before opening the chatbot
• After you type the prompt but before you send
• After the first response

Q17. What do you think will be helpful for you to be more thoughtful before using chatbots?

• Any tools you wish existed?
• What would be a realistic micro-action you could do every time before using an LLM?