Week 7 Milestone: Assumption Map + Test Cards

One idea we took from Teresa Torres’s Assumption Testing was to test the riskiest assumptions individually rather than the product as a whole. An example of how we applied this was by prioritizing assumptions based on their importance and existing evidence, then designing separate tests for Flipper’s ability to identify agenda items and detect when conversations drift off-topic. The assumptions that we drafted across the three categories were as follows:

Desirability assumptions:

  • PMs struggle to navigate their meeting schedule/productivity
  • PMs want a product that provides them with agendas/roadmaps/facilitation for meetings
  • PMs would save time from fewer distractions during meetings/more productive meetings

Viability assumptions:

  • Customers are willing to either pay for Flipper or endure ads
  • The cost of API integrations for AI features is covered by ads or Pro subscribers
  • We can find an affordable way to live-transcribe audio

Feasibility assumptions:

  • The drift signal accurately identifies off-topic conversations 
  • AI can identify agenda items during meetings
  • Customers have notes to upload and give context

From these, we drafted the following assumption map, with a few of our early priorities being placed in the upper-right quadrant. We moved the two viability concerns regarding payment into our monitor category, as opposed to the ‘test first’ category because we believe we can find evidence for this once the product is properly established, while other concerns can be addressed more immediately during this MVP stage. Furthermore, at our current scale, our AI API costs are at zero, as we’re prototyping using free models.

 

In order to test these, we’ve put together a test card for each assumption/concern.

 

Test Card: Flipper identifies the right agenda item

We believe that once Flipper v0 has an agenda loaded, it will correctly identify which agenda item is being discussed at least 80% of the time during a real meeting.

To verify, we will run Flipper on 3 meetings (live or replayed from transcripts) with PM or PM-adjacent participants who’ve each written an agenda beforehand. Flipper shows the current agenda item as the discussion moves. A teammate observes each meeting and logs the true item at every topic shift, without looking at Flipper’s output until afterward.

And measure the percentage of time slices (checked every 60 seconds) where Flipper’s displayed item matches the observer’s log.

We are right if agreement is at least 80% across all meetings, and Flipper updates within 30 seconds of a real topic change. Below 60% agreement, we treat the assumption as false.

 

Test Card: The drift signal accurately identifies off-topic conversations 

We believe the Gemini Flash 3.8 drift signal within our prototype is about as accurate as a human facilitator flagging drift. 

To verify, we will run both in the same 3 team meetings, using meetings with a written agenda. 

We will log the timestamp of every drift signal. Afterwards, a team member who wasn’t in the meeting will get the transcript and agenda, with no flags shown. They mark the stretches where the conversation left the current agenda item. 

And measure precision: the share of prototype flags that fall within or near a stretch that the labeler marked as drift. 

We are right if precision is at least 80%, with the labeler agreeing on at least 80% of stretches. 

 

Test Card: PMs want a product that provides them with agendas/roadmaps/facilitation for meetings

We believe PMs want a product that provides them with agendas/roadmaps/facilitation for meetings

To verify, we will ask five different meeting-goers about their experience following an agenda during online meetings, including their biggest roadblocks and workarounds when challenges arise. We will measure the number of meeting-goers who identify concerns related to agenda alignment and virtual meeting facilitation. 

We are right if at least three of the five meeting-goers identify concerns related to the above and their current workarounds do not sufficiently address the issue.

 

Avatar

About the author

Leave a Reply