Team Brontosaurus: Assumption Tests Report

Overview

We ran two experiments to test key assumptions about our AI usage tracking widget.

Experiment 1: Web-Based AI Usage
Tested whether users primarily interact with AI through web browser interfaces.

Experiment 2: Comfort with Prompt Tracking
Tested whether users would be comfortable installing a tool that tracks AI usage and prompt data.

Both experiments were conducted through short Google Forms distributed to frequent AI users.

——–

Experiment 1: Web-Based AI Usage

Hypothesis

We believed that users primarily interact with AI tools through web-based browser interfaces, meaning a browser-based tracking widget would capture most AI usage.

Experiment Design

Participants were asked to estimate the percentage of their AI usage across four environments:

• Web browser
• Desktop apps
• IDE plugins
• Mobile apps

They also reported how often they use AI tools and what they primarily use them for.

Form link

Participants

We recruited Stanford students who report using AI frequently.

Participants included students who use AI for:

• Coding and development
• Homework and coursework
• Brainstorming and ideation
• Writing and editing

We intentionally recruited a range of AI users in order to capture different usage patterns and ensure the sample reflected the variety of ways students interact with AI.

Artifacts

Form responses can be found here: Raw data

——–

Experiment 2: Comfort With Prompt Tracking

Hypothesis

We believed that users are comfortable installing a tool that tracks their AI usage, including prompt content.

Experiment Design

Participants were shown a description of a research tool that would track:

• Time spent using AI tools
• Number of prompts sent
• Prompt content

They were then asked:

• Whether they would install the tool
• What types of data they would feel comfortable sharing
• What privacy conditions would influence their decision

Form link

Participants

We recruited Stanford students who report using AI tools frequently.

The goal was to recruit users with different use cases, including:

• Coding and development workflows
• Coursework and homework help
• Brainstorming and creative thinking

This helped ensure we captured a range of comfort levels and privacy expectations.

Artifacts

Form responses can be found here: Raw data

——–

Final Synthesis

Learning Card 1: Web-Based AI Usage

Hypothesis, we believed that:
Users primarily interact with AI tools through web-based browser interfaces.

Observation, we observed that:
From our survey of 10 Stanford students who frequently use AI, 8 out of 10 respondents (80%) reported that they use AI tools most often through a web browser, while only 2 respondents primarily used desktop applications.

Learning & Insights, from that we learned that:
The majority of AI interactions among our target users occur in browser-based environments. This exceeds our success criteria of 60–70% browser usage, indicating that a browser-based intervention would capture most AI activity for this population.

Decisions & Actions, therefore we will:

• Proceed with building our AI usage tracking widget as a browser-based tool.
• Consider and potentially user test future support for IDE integrations for users who rely on coding workflows.

Learning Card 2: Comfort With Prompt Tracking

Hypothesis, we believed that:
Users are comfortable installing a tool that tracks their AI usage, including prompt content.

Observation, we observed that:
Out of 5 respondents:

• 1 respondent would install the tool without hesitation
• 2 respondents would install it with conditions
• 1 respondent was unsure
• 1 respondent said they would probably not install it

This means 3 out of 5 respondents (60%) expressed comfort or conditional comfort installing the tool, which falls below our success threshold of 70%.

However, when participants were asked if they would install the tool if prompt content were anonymized and deletable, the responses shifted:

• 3 respondents said Yes
• 2 respondents said Maybe
• 0 respondents said No

Learning & Insights, from that we learned that:

Users are not inherently opposed to AI usage tracking, but privacy and control over prompt data are major concerns. Adoption appears highly dependent on clear safeguards such as anonymization, local storage, and the ability to delete collected data.

Decisions & Actions, therefore we will:

• Clearly communicate what data is collected and how it is used.
• Explore local storage options for prompt tracking where possible.