Learning cards
| Test | Prototype test |
| What we expected | People would rather use our Slack Bot over other software that organize and summarize work meetings |
| What happened | Interviewee #1 (B2B SaaS Startup PM:
Somewhat enthusiastic about our product. Appreciates the status update feature: likes how the bot tells the user why something is blocked and its importance. Has worries about accuracy and some concern about integration. Wouldn’t be fully ready to replace existing tools they are using. Interviewee #2 (Healthcare Startup): Finds the status summary to be the most helpful, especially for managing the project day to day. In addition, finds the document feature interesting; only big concern is over documentation approval and privacy. Their company uses Slack frequently, so consolidating information into our bot would be really helpful. Would also be willing to try our product with existing tools they are currently using. |
| What we’re changing | Improve integration, it’s good to consider that ProjectGenie can be adopted into workflow with other existing tools. Emphasize our product’s ability to summarize and build user trust with clearer privacy measures and approval controls for any AI-generated content. |
| Test | Price test with $20 monthly subscription |
| What we expected | We expected at least one third of test users to be willing to pay $20/month for our product |
| What happened | ⅔ interviewees said they would pay $20/month for our product. |
| What we’re changing | We choose $20/month as the price for our product. |
| Test | Fake door test |
| What we expected | at least 10% of people who visit the site sign up to try our product. |
| What happened | After posting on reddit and Slack channels with PMs, out of 12 people who visited, 1 of them signed up by giving us their emails. |
| What we’re changing | We’ll change how we communicate our value proposition on our website and see how it changes the visit to sign up ratio. |
Evals.
Some context: Our current prototype is a clickable Slack-style demo built with Lovable, and all of its content comes from a mock data file describing one fictional project. The bot’s answers and stakeholder summaries are scripted responses chosen by keyword matching rather than generated by a language model, so these evals test the intended behavior and expose where the scripted approach breaks, and we plan to replace the scripted logic with real LLM calls that draw on connected project sources as the product develops. The code checks called the bot’s response function directly with each input. The LLM judge row was graded by Claude using the fixed rubric in the table, not by a separate automated API call. We will work on implementing actual calls to LLMs as necessary in future iterations.
| Input | Good output | How we judge | Result | What we’re changing |
| “Has the checkout redesign ticket been completed?” (Engineer role) | Says CHK-142 is not done, gives its current status, and cites the ticket | Code: answer contains “CHK-142” and “In Review”, and the citations include CHK-142 | Pass | Nothing for this case. We will keep it as a regression test once a real model replaces the script. |
| “Is CHK-138 done?” (PM role) | Answers about CHK-138, the wallet ticket, which is blocked | Code: answer mentions “CHK-138” and is not the CHK-142 answer | Fail | Replace keyword matching with a model that reads the full question and looks up the ticket it names. |
| “What’s blocking launch?” (Leadership role) | Names the main blocker in plain language and explains the business impact | LLM judge: names the blocker, states the business impact, uses no ticket IDs or jargon, and matches the project data | Pass | Mention every current blocker, including the A/B tracking work, and say what the mitigation is instead of “Mitigation is in motion.” Add both points to the rubric. |
| Draft summaries, Sales audience, with the prompt “Weekly update on checkout redesign progress and risks” | A customer-ready update in plain language that answers the PO flow question and could be forwarded without edits | Human: a teammate playing a sales rep answers yes or no to three questions. Is it clear to a customer? Does it answer the PO flow question? Would they forward it without editing? | Fail | Remove internal notes from customer-facing summaries. The line about Marcus’s one-pager is meant for the team, so a rep would have to delete it before sending. |
| “Why does this project matter?” (Engineer role) | Explains the business goal, why this work comes before other projects, and how the engineer’s own tickets contribute | Human: a teammate playing an engineer scores each point from 1 to 3. Passing means at least 2 on every point. | Fail | Add a comparison with other projects and link the answer to the asker’s own tickets. The current answer gives the 12% conversion goal and research data but never says why this outranks other work. |
Team Product OKRs for the rest of the quarter. One objective, 3–5 key results.
Objective – Successfully consolidate workflow for people working in product management.
Key results
KR1: 10 test users use our product for a week for project progress tracking.
KR2: Over 50% of test users report needing fewer meetings with our product.
KR3: 20% of test users state that they would use the product again.
Idea from this week’s reading
OKRs should combine an “inspirational” objective with measurable key results. We applied this by setting one clear team objective around consolidating product management workflows and defining three measurable key results to track user adoption, meeting reduction, and willingness to reuse the product. We also set KR2’s target with Christina’s 50% confidence rule.
