Team 4 – Week 8 Milestone: Test Results + Metrics + OKRs

Learning Card 1 – Flipper identifies the right agenda item

We believed Once Flipper had an agenda loaded, it would show the item the team was actually discussing at least 80% of the time at one-minute checkpoints and update within 30 seconds of a topic change.
We observed Meeting 1 (logo design): The write-up reports 9 of 21 checkpoints matched (43%), or 13 of 21 (62%) if four ambiguous checkpoints are counted as matches. Flipper skipped logo selection, took 3 minutes 49 seconds to switch to color, and switched to next steps in 9 seconds.

Meeting 2 (roadmap, onboarding, and budget): The team explicitly introduced both new items. The aligned stopwatch and transcript put the update delays at about 41 seconds and 59 seconds, respectively. Two likely mismatches put agreement at 5 of 7 checkpoints at best (71%) if both hold. The 7:00 checkpoint is sensitive to a few seconds of clock alignment.

Meeting 3 (evaluation planning): The team switched to eval design at about 1:48. Flipper still showed the first item at 4:15, when the timer prompted the team to move on. The team manually skipped ahead, so eval design logged 0 minutes even though it took about 7½ minutes of the meeting. We lacked a complete checkpoint log, and the manual click ended the period we could use to assess automatic switching.

From that we learned The test did not pass. Across the three meetings, Flipper skipped agenda items, updated too late, or relied on the team to advance the agenda manually. We learned that reliable agenda tracking depends on recognizing when the team’s objective changes, even if the subject stays the same. Timing matters too: a correct label is less useful once most of the discussion has already happened. Manual controls can keep the meeting moving, but the time spent on a skipped item can still go unrecorded.
Therefore we will Improve agenda tracking by giving each item a short goal, as suggested in Meeting 1, so Flipper can tell when the team moves from reviewing options to making a decision. If Flipper misses a switch, the host can use the “Hey Flipper” wake phrase proposed in Meeting 3 to move to the right item without opening the agenda editor. For example, “Hey Flipper, switch to eval design and give it five more minutes” would update the current item and rebalance the remaining agenda time. The host could also correct the log to capture time the team already spent on that item.

Learning Card 2 – The drift signal accurately identifies off topic conversations

We believed Flipper’s Gemini Flash 3.8 drift signal would be about as accurate as a human facilitator at identifying when a conversation left the current agenda item. We expected at least 80% of Flipper’s alerts to fall within or near a stretch an independent reviewer marked as drift, with agreement on at least 80% of stretches.
We observed In Meeting 1, the team returned to changing logo shapes for 3 minutes 27 seconds after moving to next steps. Flipper gave a facilitation nudge about generating more variants, but no drift alert was recorded. 

In Meeting 2, Flipper’s only recorded drift alert fired 53 seconds into the winter-travel tangent, while the team was still off-topic. 

In Meeting 3, the team moved from defining evaluation criteria to designing the evaluation process and spent about 7½ minutes there. No drift alert appeared in the screenshots. The write-up classified this as an intentional move to the next planned item that Flipper missed.

From that we learned Flipper met the observed precision target, but the overall test was inconclusive. Its only recorded alert came during Meeting 2’s winter-travel tangent (1 of 1, or 100%). Meeting 1 showed what that score leaves out: the team revisited logo shapes for more than three minutes, and the missed detour never entered the calculation. Together, Meetings 1 and 3 show why the team’s intent matters. Returning to logo shapes pulled the team away from next steps, while moving into evaluation design advanced the planned agenda. Without an overall agreement score, we couldn’t verify the 80% agreement target or conclude that Flipper performed about as well as a human facilitator.
Therefore we will Improve how Flipper uses the recent discussion and written agenda to decide when to flag drift. When the team deliberately starts the next item, Flipper will update the agenda. When a side conversation continues, whether it revisits an earlier item or goes entirely off agenda, Flipper will offer a quiet prompt allowing the team to choose whether to resume the discussion or continue to the next agenda item. 

Learning Card 3 – Desirability Test

We believed PMs would want a product that helps them plan agendas and facilitate virtual meetings. We would pass the test if at least 3 of 5 interviewees reported having an agenda alignment or facilitation problem that their current workarounds didn’t fully solve.
We observed At least 3 of 5 met both criteria, with a likely fourth. Sid said written agendas and Slack recaps still left meeting goals and decision owners unclear. Ellie relied on hosts to redirect crowded calls, and unresolved questions sometimes led to another meeting. Katherine said documents, notes, and follow-up emails helped with organization but didn’t address participation problems. Addie still struggled to track time and redirect discussion manually, though she wasn’t asked directly whether her workaround was sufficient. Peter reported rambling and uneven airtime, but we couldn’t tell whether his manual nudges and time reminders fell short.
From that we learned The test passed. Several participants already had agendas or ways to share them, yet still depended on someone to clarify the goal, redirect the conversation, or make room for people to speak. That gives us a reason to test live facilitation. Whether full-time PMs would adopt Flipper remains open. Both PM interviewees were interns, and the interviews measured problems and workarounds.
Therefore we will Start testing Flipper with full-time PMs who run recurring meetings. Specifically, we’ll test whether stating the meeting goal and who owns the decision up front, along with quiet agenda and drift prompts during the call, helps hosts keep the discussion on track. After each meeting, we’ll ask hosts for specific examples of when a prompt helped, was ignored, or interrupted the discussion. We’ll then give them the option to use Flipper again at their next meeting and record whether they do.

Prototype Changes

For this milestone, we built the first version of Flipper and tested it in three live team meetings. flipper-mvp.vercel.app

Current features

  1. Agenda setup. Teams can load a written agenda or generate one from meeting notes, with a planned duration for each item.
  2. Live transcription. Flipper captures meeting audio and produces a timestamped transcript as the conversation unfolds.
  3. Live agenda tracking. Flipper displays the current agenda item, tracks elapsed time, and prompts the host when an item runs over its allotted time. The host can also advance the agenda manually.
  4. Drift signal alerts. Flipper alerts participants when a discussion goes off-topic, reminding them to stay on track.
  5. Meeting follow-up. After the meeting, Flipper lists decisions and to-dos. Supporting transcript excerpts show where each task came from.

Features we plan to add or improve

  1. Improve agenda recognition. Give each item a short goal so Flipper can recognize when the team’s objective changes in situations where the same language is used between topics. 
  2. Add “Hey Flipper” voice controls. Let hosts switch items, extend a discussion, or rebalance the remaining agenda time without opening the agenda editor.
  3. Improve drift signal prompts. Use the agenda and recent discussion to distinguish a planned transition from a sustained detour. Flipper should update the agenda when the team deliberately moves on. If a side conversation continues, it should quietly send a prompt asking whether the team wants to keep discussing it or return to the current item.

Metrics Tree  

Our North Star metric is the number of productive Flipper-assisted meetings per week, defined as meetings that end on time with all planned agenda items covered. This measures both product adoption and the value Flipper delivers.

L1 Metrics – Primary Drivers:

  1. Flipper-assisted meetings per week: How frequently users incorporate Flipper into their meetings.
  2. Productive meeting rate: The percentage of Flipper-assisted meetings that end on time with all agenda items covered.

L2 Metrics – Supporting Drivers:

  • Usage: Weekly active meeting hosts and average Flipper-assisted meetings per host.
  • Meeting success: Percentage of meetings ending on time and percentage of those meetings covering all agenda items.

L3 Metrics – Feature-Level Drivers:

  • Adoption: First-meeting activation rate, repeat usage rate, and agenda setup completion rate.
  • AI facilitation performance: Agenda identification accuracy, drift alert precision, drift detection speed, and actionable reminder rate.

Guardrail Metrics: We will also monitor false drift alerts, user satisfaction, and willingness to reuse Flipper to ensure the product improves meeting effectiveness without creating unnecessary interruptions.

This tree connects individual product features to measurable meeting outcomes. While L1 and L2 metrics track overall usage and productivity, L3 metrics help identify which aspects of Flipper’s functionality contribute to those outcomes. This allows us to prioritize improvements based on whether they meaningfully help users run more focused and productive meetings.

Evals

We will evaluate Flipper using eight simulated meeting transcripts, each designed to test agenda identification or drift detection. Every test will use the same three-item agenda: (1) Product Roadmap, (2) Customer Onboarding, (3) Budget Allocation.

Method: We will write short, realistic meeting transcripts and provide both the agenda and transcript to the model through the same AI processing pipeline used by Flipper v0. Where possible, transcripts will be fed sequentially to simulate a live meeting and test how quickly Flipper responds to topic changes. 

For each test, we will specify the expected output beforehand: which agenda item Flipper should identify and whether it should flag the conversation as off-topic. We will then compare the actual response to this expectation.

Test Cases (each followed by its expected AI response):

  1. Clear agenda match: “Let’s prioritize features for next quarter’s release.”
    Expected: Product Roadmap, no drift.
  2. Semantic recognition: “Users keep abandoning the account setup process.”
    Expected: Customer Onboarding, no drift.
  3. Budget identification: “How should we distribute our remaining $20,000?”
    Expected: Budget Allocation, no drift.
  4. Agenda transition: “That covers our roadmap. Let’s discuss next quarter’s budget.”
    Expected: Flipper identifies Budget Allocation at the first check after the transition and switches the agenda from Product Roadmap within 60 seconds.
  5. Overlapping topics: “We’re prioritizing onboarding improvements for the next release.”
    Expected: Customer Onboarding, no drift.
  6. Off-topic discussion: Participants discuss vacation plans for about 75 seconds during a roadmap discussion.
    Expected: Drift flagged within 60 seconds of the tangent starting.
  7. Brief tangent: Someone makes a five-second unrelated joke before returning to the roadmap.
    Expected: No drift alert.
  8. Returning to topic: “Anyway, back to our roadmap and feature priorities.”
    Expected: Drift flagged during the retreat tangent (about 75 seconds), then Product Roadmap with drift cleared once the talk returns.

Results and Learnings: We will mark each test pass/fail based on whether Flipper produces the expected response, calculate the overall pass rate (X/8), and document any unexpected behavior.

How we ran it: each transcript was split into 30-second windows and fed to gemini-3.1-flash-lite, the model Flipper v0 uses, through Flipper’s prompt and its topic and drift timing rules. Every case ran 3 times. A case passes when (A) the model’s answer is right for every 30-second window, (B) the agenda switches within 60 seconds of each real transition with no false switches, and (C) off-topic talk of 60 seconds or more is flagged within 60 seconds with no false alarms.

Evals Results

6 of 8 cases passed before the fix. Both failures were off-topic detection (cases 6 and 8). After the fix, all 8 cases pass 3 of 3 runs, so the prompt change caused no regressions.

Case Expected Before fix (3 runs) After fix (3 runs)
1 – Clear agenda match Product Roadmap, no drift Pass 3/3 Pass 3/3
2 – Semantic recognition Customer Onboarding, no drift Pass 3/3 Pass 3/3
3 – Budget identification Budget Allocation, no drift Pass 3/3 Pass 3/3
4 – Agenda transition Switch to Budget Allocation within 60 s Pass 3/3 Pass 3/3
5 – Overlapping topics Customer Onboarding, no drift Pass 3/3 Pass 3/3
6 – Off-topic discussion Drift flagged within 60 s Fail 0/3 Pass 3/3
7 – Brief tangent No drift alert Pass 3/3 Pass 3/3
8 – Returning to topic Drift flagged, then cleared when talk returns Fail 0/3 Pass 3/3

Timing and speed: agenda switches appeared 31 to 59 seconds after the real transition, and drift alerts appeared 37 to 52 seconds after the tangent began. There were no false drift alerts and no false agenda switches in any run. The median model call took 1.2 seconds (slowest 5% about 6 seconds), with about 1,000 input and 80 output tokens per call.

Learning 1 (an assumption that turned out wrong): we assumed the model’s off-topic flag and its nudge text would always agree. They did not. It wrote a correct drift nudge while reporting offTopic as false, so the alert was hidden. Decision: tighten the prompt so the two must match (see Prototype Changes).

Learning 2: Flipper reacts 30 to 60 seconds after a change by design. The 5-second joke in case 7 raised no alert, and tangents under 30 seconds are never flagged. Decision: keep the 30-second hold, since it produced zero false alerts and zero false switches.

Learning 3: Gemini sometimes returned 503 or 504 errors or took over 30 seconds to answer. Decision: the eval now retries these errors so an outage is not counted as a model failure, and Flipper keeps its timer-based fallback for ticks where the model does not answer.

Prototype Changes Continued

Specific failures and the resulting modifications to our prompts, classification logic, or drift detection thresholds

Failure found: in cases 6 and 8, Flipper’s model wrote the right nudge (“We’ve drifted to holiday plans. Let’s park that and return to the product roadmap.”) but set offTopic to false and kept the current agenda item as the topic. The app only shows a drift alert when offTopic is true, so the alert never appeared (0 of 3 runs in both cases).

Change (committed to the app, src/lib/facilitation/providers.ts): we added two rules to Flipper’s prompt. First, the offTopic rule now lists vacations as an example and adds “If offTopic is true, topicItemId MUST be null” (in both the main prompt and the topic-only prompt). Second, the refocus rule now says the nudge is written ONLY when offTopic is true, and that refocus MUST be null when offTopic is false.

Not changed: the 30-second check interval and the 30-second hold before an agenda switch or drift alert. Across all runs there were no false drift alerts and no false agenda switches.

Team Product OKRs

Objective: Make Flipper the meeting facilitator PMs trust to keep their virtual meetings focused and productive.

Key Results:

  1. Accuracy: Pass at least 7 of our 8 evals, and Flipper picks the right agenda item at least 80% of the time in real meetings.
  2. Drift detection: At least 80% of drift alerts are correct, with no false alerts for brief tangents under 10 seconds.
  3. Real usage: 3 PMs or PM-adjacent users run Flipper in at least 2 real meetings each before Dec 1.
  4. Meetings stay on track: At least 70% of meetings run with Flipper end on time with every agenda item covered.
  5. Value: At least 2 of 3 test users say in a post-meeting survey that Flipper made their meeting more focused and that they’d use it again.

Idea From Reading

From Wodtke’s Introduction to OKRs (Ch. 2), we used the rule that an Objective should be inspirational and contain no numbers, since a goal like “double the users” is really a Key Result. Our objective states the outcome in words, and the numbers live in our five Key Results.

Assumption and Adaptation

We assumed a participant could join a virtual meeting and run Flipper on the same device. Ideally, users could speak through a laptop’s microphone for the call while Flipper captured the conversation. This assumption turned out wrong because Flipper repeatedly crashed in this setup which made it difficult to run our tests. However, we adapted by keeping the meeting on the laptop and running Flipper on a phone beside the laptop speaker. 

Avatar

About the author

Leave a Reply