Milestone 8: Test Results + Metrics + OKRs – Team 7

Team Members: Diego Sanchez Tirado, Felicia Yan, Isaac Were, Itai Reingold-Nutman, Janus Tsen, Victor Ahona, Carolyn Lee

Learning Cards

1. Responding to system pings – People will be willing to meaningfully respond to pings from the system

  • Test: We want to test if people will be willing to respond to pings from the system throughout their workdays. We plan to test this via interviews with PMs where we can hypothetically ask them how they would feel about such a system, and what if anything would make it so that the system is acceptable to them in their day-to-day work. 
  • Hypothesis: We believed that employees would answer pings under specific conditions and incentives: Namely, (1) that the pings are not “too” frequent such that they disturb their day-to-day activities, (2) the pings do not require immediate attention and can be answered during some grace period, (3) the pings almost always pertain to areas of knowledge and expertise that the employee can speak to (ie: the pings very rarely irrelevant, unanswerable, or out-of-scope questions), and (4) their response rate and speed are recorded and available as a record for their higher-ups. 
  • Observation: We observed that interviewees were open to the system and responding to pings, but there were some factors to consider:
    • Frequency: 3+ distinct broader pings from the same person in a day was already seen as disruptive, and the ceiling across all pings was roughly 10-15 per day. Also, higher-level employees would likely receive more pings and may respond less.
    • Format: One or two broader messages (“here’s what I’m thinking about, here are some areas to explore, what are your thoughts?”) were strongly preferred over 5-10 narrow, specific questions, since they allow a single response with less context switching.
    • Channel: One-on-one messages are better than group chats. In group chats, pings get buried, and people assume someone else will answer.
    • Attribution: Our participant suggested framing pings as “asked on behalf of [PM name]” rather than anonymously. This adds a sense of community and mild accountability, while keeping it from feeling like a direct demand from the PM.
    • Grace period (Hypothesis 2): Giving people time to answer reduces pressure, but explicit due dates or grace periods made it feel like an assignment. A long window also slows the system down for the person who asked. So there are trade-offs, and we should run tests to see how different levels of urgency in pings impact response rate.
    • Relevance (Hypothesis 3): Participants treated relevance as a core requirement and something to optimize for.
    • Tracking (Hypothesis 4): Our participant was wary of using response rate as a metric early on. It would be high variance given differences in ping volume and type, and it could incentivize people to use their own AI to answer pings, creating an AI-answering-AI loop. If tracked, managers should see only team-level aggregates rather than individuals.
    • Rerouting: Participants wanted the ability to forward a ping to a more relevant person, since people have a clear sense of what they own.
  • Learnings and Insights: From that we learned that willingness to respond is less about incentives and more about low friction, relevance, and social framing. Our hypotheses about low frequency and relevance held up, and we now have rough numbers to design around. The grace period hypothesis needs refinement: people want time to respond, but the framing should be “you’re helping the team” rather than “this is due.” But, the hypothesis that recorded response rates would be visible to higher-ups was not supported, since it risks gaming and feels like surveillance, so aggregate team-level visibility is a better fit. We also uncovered two things we hadn’t hypothesized: people want to bounce pings to the right owner, and named, one-on-one pings outperform anonymous or group ones.
  • Decisions and Actions: Therefore, we will cap pings at roughly 2 broader pings per person per day, and make sure to stay well under 10-15 total, and bundle related questions into one open-ended message instead of many narrow ones. We will deliver pings as one-on-one DMs attributed as “on behalf of [PM name],” and build a “refer to someone else” action so recipients can redirect a ping instead of writing a new message. In the ping UI, we will avoid explicit deadlines and frame requests around helping the team. We will also A/B test grace-period messaging to balance response speed against pressure. We will not use individual response rate as an early success metric, and if we track it later, we will show only team-level aggregates to managers. Finally, we will prioritize relevance as a core optimization target.

2. Visibility of work – People are comfortable having their in-progress workstreams visible to other teams

  • Test: We will test whether a person is comfortable with having their in progress work visible to others. To do so, we will ask for sample work from the interviewee and their feelings towards sharing this to colleagues on other teams. Then, we share the sample work with a different interviewee and ask what information they would find most useful. 
  • Hypothesis: We believed that the interviewee will be comfortable sharing workstreams if they see others doing the same.  
  • Observation: We observed that our participant was generally comfortable sharing her in progress work with others, particularly if they were colleagues that she already knew. Though, her willingness to share depended more on who would have access to the information instead of whether other people were also sharing. 
  • Learnings and Insights: From that we learned that visibility of work is not necessarily an issue of reciprocity but one of trust. People are more comfortable with sharing unfinished work if they know its with others that already understand the context of the projects. 
  • Decisions and Actions: Therefore, we will explore ways for Index to prioritize sharing information with relevant or adjacent teams instead of just making all work accessible by default.

3. Workstream discovery – PMs value proactively discovering relevant workstreams across the organization, even when they weren’t previously aware of them or don’t have an immediate need to know about it 

  • Test: We will test whether PMs actually want to be told about workstreams they didn’t ask about. Two mechanisms. First, in interviews, show each PM a short list of 3–5 real in-progress workstreams from other teams (pulled from the sample work gathered in card 2) and ask which, if any, they would have wanted to know about and why. Secondly, over one week, send each PM one “did you know team X is working on Y” message a day and record whether they open it, reply, forward it, or ignore it. Ensure 1 or 2 are real near-duplicates of their own work – strongest test of value.
  • Hypothesis: We believed that PMs will value proactive discovery only under narrow conditions:
    1. Workstream overlaps with or blocks something they own.
    2. It arrives early enough to change a decision, not after the fact
    3. It comes at low volume, either around one a day or batched weekly so it reads as a heads-up.
    4. It includes who to talk to, so the next step is one message, not a search.
  • Observation: We observed that PMs responded positively as soon as we showed them the workstream list, and the interviews opened on a strong sense that this was valuable. A good number of PMs had no idea what other teams were working on or how far along they were. They were most engaged, often fascinated, when a workstream was similar to their own, and these near-duplicates got the most reaction and follow-up questions.
  • Learnings and Insights: From that we learned that PMs want to know about other teams’ work more broadly than our narrow hypothesis assumed. The visibility gap is real: PMs aren’t avoiding this information, they just have no easy way to find it. Overlap is still where the most value sits, because spotting similar work early can prevent duplicated effort, open up reuse and connect PMs who should be talking. Interest alone doesn’t prove PMs will act on what they learn, though, so timing, volume and a named contact still need to be tested.
  • Decisions and Actions: Therefore, we will keep proactive workstream discovery as a core feature and design it around overlap first, putting the most similar work at the top.

Evals

The primary task that AI supports in our product is asking about workstreams and receiving a reliable answer or having action taken on your behalf to find the missing information. Given a PM’s question, does Index answer from the knowledge base when it has current, sourced information and identify the right teammate with a clear question when it doesn’t? Our 8 evals for the feature are below: 

 

Test case / scenario Input Expected Output Scoring Prototype eval results
User asking about work / person that is not in the system  Asks Index about a workstream that is not mentioned anywhere in existing docs Index acknowledges that context is missing and suggests next steps Accuracy – does not hallucinate answers or citations Fundamental items are included such that only <20% of people request a new feature. 
Correctly identifies relevant colleague when a user asks Asks Index about a question that can be answered by a colleague Index provides a list of relevant people that can answer the question given Accuracy – links the correct person who is able to answer the question the user asked Miss rate for appropriate colleague identification is <15%
A user sends out multiple questions to index to find information about relevant workstreams User submits several questions about different projects or teams in a single query Index processes the questions in parallel, retrieves relevant context for each, and returns organized answers without requiring separate queries Speed – Index responds to multiple questions efficiently, with minimal increase in response time compared to a single query Ability to do parallel tasks
User asks about work where documents are relatively old Ask question about files for feature colleague is working on Index summarizes content but also flags the age of the content Accuracy – includes acknowledgment of potential stale content, minimize false positives for content that is likely still true (e.g. no longer in development, already shipped) Surfaces both old and new items but ranks based on relevance, not just age. Reasonably accurate–not hard metric. 
System notifies user to answer a teammate’s question Ask a question that can’t be answered by existing files alone Index initiates a ping only when necessary, and notification is concise, readable, and actionable Accuracy – does not ping unnecessarily, for inconsequential questions Notification actuates on command. 
Conflicting sources – multiple files give different answers Ask a question about sources with contradicting information Surface the source that is the most accurate based on  recency Accuracy – pass if the older claim is never brought up as the current plan  Output is solely based on research gathered and is coherent with initial requests. 

Change: make uncertainty visible, show reasoning behind claims (beyond recency, also who’s the current owner, level of confirmation, etc)

Rerouting and reusing answers – instead of pinging many times for the same question, information should be saved once it’s received User 1’s ping is sent, user provides response → later, user 2 asks the same question Saves the response into knowledge base, and returns updated knowledge in re-asked question Accuracy – successfully incorporates new knowledge and returns it to other users 

Speed – knowledge becomes queryable quickly, within < 30 sec

Back and forth is limited to crucial items. Search is comprehensive enough to yield answer upon minimal prompting without use re-routing. 
Respects visibility permissions with who can see in-progress work User on an unrelated team asks about a workstream shared only with adjacent teams Index doesn’t reveal the restricted details, makes visible that the work exists but access is limited Accuracy – 0 restricted details appear in the answer Always asks for permission when retrieving information. 

OKRs

Objective: Establish real value by making cross-team collaboration easy, seamless, and less frustrating for users

  • KR1: Reduce time spent in cross-team alignment meetings by 10%
  • KR2: 50% of PMs who try Index in our pilot return to use it again the following week (weekly retention)
  • KR3: Cut time to get context about other teams by 50% (estimate based on user surveys)
  • KR4: Achieve 70% of pings answered within 24 hours

One line from the reading: From AI Evals Explained Simply, we applied the idea of scenario-based testing with measurable success criteria to our eight evals for Index, focusing on real user interactions and evaluating performance based on accuracy, speed, and reliability.

Avatar

About the author

Leave a Reply