Team OKR
Objective: Make every meeting more productive and focused
KR
- 60% of interruptions are accepted by user
- 30% of first time users, continue using it for recurring meetings for the next 3 months
- Reduce meeting time by 10%
- Local model flags off topic discussion within <3 seconds
Evals:
Task: moderate a meeting by popping up at the correct times
Scenarios:
- Someone asks a question that no one answers
- People go off track
- Two agenda items have not been covered and there are 5 minutes left
- Someone asks a question and it is immediately answered
- People wander off agenda, but the discussion is still tangent of the meeting items
- Someone asks a question that was asked in a meeting last week
Scoring the output:
- Accuracy: 95% correct interjections
- Reliability:
- Speed: Within 30 seconds since going off track or an unanswered question
- Cost: $.01 per pop up
Scenarios:
- Someone asks a question that no one answers
- Input: Give Quicken a transcript where a questions asked is not answered/Quicken is analyzing the transcript in real time
- Expected Output: Quicken pops up to point out the question that was not answered
- Scoring Criteria:
- Quicken notifies after 60s that a question has not been answered after the user dismisses the question, Quicken will not notify the same question again.
- Popup names the correct missed question
- People go off track
- Input: Start Quicken in a meeting with the agenda items filled out, they spend >5 minutes talking about hobbies at a random point.
- Expected Output: Quicken pops up to remind people to get back on track
- Scoring Criteria:
- pops up within 1 min to notify the user that the conversation is not related to the topic
- doesn’t start until they start discussing the first agenda time. Should avoid greetings
- Two agenda items have not been covered and there are 5 minutes left
- Input: a meeting transcript with an agenda, the meeting end time, the current time, and the agenda items that have been covered so far
- Expected Output: Quicken pops up with a message to cover the most important agenda items
- Scoring Criteria:
- Correctly identifies which agenda items have not been covered
- Correctly recognizes that there are only five minutes left in the meeting
- Clearly identifies which agenda items still need attention
- Prioritizes which agenda item to cover based on importance
- Does not incorrectly flag agenda items that have already been covered
- Someone asks a question and it is immediately answered
- Input: a meeting transcript in which someone asks a question and another participant answers it shortly after
- Expected Output: Quicken does not pop up
- Scoring Criteria:
- Correctly recognizes that the question has been answered
- Quicken does not pop up
- Does not flag the question as unresolved later in the meeting
- People wander off agenda, but the discussion is still tangent of the meeting items
- Input: A transcript with agenda items. During one of the agenda items, they start discussing something else that turns out to be blocking this agenda item.
- Expected Output: Quicken does not intervene
- Scoring Criteria:
- Responds in <30s whether or not it should intervene or not.
- Treats tangent conversation as relevant and not off topic, 70% of tangent conversations are recorded as relevant
- Someone asks a question that was asked in a meeting last week
- Input: a current meeting transcript and the transcription from the previous week’s agenda
- Expected Output: Quicken pops up with a callback to the answer in the transcript from last week, referencing the question and answer
- Scoring Criteria:
- Correctly identifies that the question was previously asked
- Retrieves the correct answer from the previous transcript
- Provides the relevant context, inducing when the question was previously discussed
- Does not confuse similar questions with different answers
- Does not hallucinate information from previous meeting
- Respond within 30 seconds of identifying the repeated question
Link to the Reading
We used Hamel Husain’s Level 1 idea of breaking an LLM feature into scenarios and testing each with assertions when iterating on our v0 prototype to test assumption three because Quicken’s moderator is split into 42 different scenarios and we scored each against targeted checks.
