Learning Card 1:
| Test | Show 2-5 IT administrators a prototype demonstrating AI-generated Slack messages, automated follow-ups, and interpretation of approver responses. Ask them to choose which actions they would allow AI to perform autonomously versus requiring human approval.
Prototype v0: https://cs177-doorman-v0.vercel.app/ |
| What we expected | Okay with all three: automated follow-ups, access to Slack, and access to company policy |
| What happened | – Not ok with having access to all Slack messages in a workspace, a designated channel would be fine
– Company policy is out on the open internet for Stanford, so it’s okay to use – Automated follow ups need to disclose that it’s an AI |
| What we’re changing | – Only allow access to certain Slack channels, or have a dedicated Slack channel
– Look at Stanford Administrative Guide for IT admin policy – Make sure to disclose it’s an AI message if the product reaches out to someone |
Learning Card 2:
| Test | Show 2-5 IT managers or budget owners a landing page with the product’s capabilities and a proposed monthly subscription price. Offer an opportunity to request a paid pilot, and conduct follow-up interviews to assess purchasing intent.
Landing Page: https://joshuatikare.github.io/cs177-doorman-landing-page/ |
| What we expected | They would pay for the product at the price listed |
| What happened | – At Stanford Tech Desk, most of the context is in the ticket
– Landing page is good, does describe the product – Not their decision on whether to buy it – Control access on ServiceNow, biggest issue on Confluence, because they don’t control it – UIT handles tickets, only student tickets are passed to Tech Desk, they have full control over their tickets, and context is rarely missing – Comments on pricing, below |
| What we’re changing | Price could be $35/seat. Slack is $15/seat. HubSpot is $10/seat. ChatGPT is $25/seat. $30-40 range is good starting out since not many people on the team need the tool, then charge more once you refine the product. Can also have bulk discount for adding more seats |
Learning Card 3:
| Test | Give 2-5 IT administrators a simulated access request. Have them complete the workflow using their existing process with the help of Doorman.
Prototype v1: https://cs177-doorman.vercel.app/ |
| What we expected | 20 seconds to approve a request |
| What happened | – 45 s to approve
– 1 time LLM didn’t load because in Incognito tab – At help desk have to troubleshoot with students, so it might take longer to get all the context |
| What we’re changing | Add automated follow ups when people don’t respond to reduce time to complete ticket |
Assumption that turned out wrong and how you adapted:
Our assumption for our third test turned out wrong. We thought it would be much quicker to approve a request, but we realized (a) the LLM takes some time to load, and (b) there is a good amount of information for the admin to scan through. The admins also said the process would involve outreach and back-and-forth with the requester. We will adapt by adding automated follow up messages when people don’t respond to expedite this process.
Evals:
Speed: all under 1s, not a concern
| Input | Good output | How we judge | Result |
| Ticket requesting access to figma editor for mobile redesign, manager approved | Approve as editor | Code:
Pass if decision = approve AND level = editor |
Pass: Approved Figma access at the editor level, as requested and authorized. |
| Github write access request approved by tech lead | Approve as write | Code:
Pass if decision = approve AND level = write |
Pass: Approved GitHub access at the write level, matching the tech lead’s approval. |
| Asks for AWS admin access but needs to look at images | Approve as read-only | Code:
Pass if level = read-only |
Partial pass:
Correctly reduced AWS access to read-only in some cases, but did not do so consistently. |
| Snowflake write access saying Tom approves, but Slack context from Tom says to hold | Decline or hold and mention Tom’s message | Code, Human:
Teammate rates 1-5 Pass if decision=decline and rating > 80% |
Partial pass:
Code pass; human rating 4/5: |
| 2 week AWS admin access request, company policy max is 8 hours | Approve for 8 hours and reference policy docs | Code: Pass if decision=approve and limit=8 hours |
Pass: Approved AWS access for 8 hours, within the company policy limit. |
| Someone from marketing requests GitHub write access to edit page copy | Defer to human, but mention context that the person is not an engineer | Human, LLM judge:
Teammate and LLM rate 1-5 Pass if > 4 |
Fail: Did not meet the criteria for human escalation with an explanation of the requester’s non-engineering role. |
| Contractor requests snowflake write access, policy forbids it, but manager asked for it on slack | Decline and suggest staging | Code and LLM judge:
Pass if decision=decline and staging or suitable alternative suggested |
Fail: Did not meet the combined requirement to decline access and suggest staging or another suitable alternative. |
| Finance employee requests full access to Stripe | Approve as reporting only and reference relevant policy | LLM judge:
Pass if judge rules that the policy is relevant and approval status is correct |
Pass: Approved reporting-only Stripe access and referenced a relevant policy. |
| Missing context “github write access pls” | Set status to on-hold | Code:
Pass if status = on hold |
Pass: Placed the request on hold because it lacked sufficient context. |
What we’re changing based on evals:
Adjusting the prompt to pass as many of the two evals it failed and two it partially passed as possible.
- Grant the minimum access needed for the task. E.g. For example, grant read-only AWS access when someone only needs to view images.
- Explain conflicting approvals. E.g. When a ticket claims Tom approved access but his Slack message says to hold, cite that message in the decision.
- Escalate requests that fall outside standard role permissions. E.g. When someone in marketing requests GitHub write access to edit page copy, route the request to a human and explain the requester’s role and what they need to do.
- Enforce policy and suggest a permitted alternative. E.g. Decline a contractor’s Snowflake write request when policy forbids it, even if a manager requests it in Slack. Suggest staging if policy permits it.
OKRs:
O: Objective
- Save time for PMs waiting on tool access permission with AI IAM software
KR: Key Results
- 10 demo tickets resolved
- 50% time reduction to resolve ticket (using our tool vs human alone)
- 90% correct approval suggestion
Reading insight:
We used Gupta’s idea of defining what “good” looks like in advance when designing our evals: each test case had explicit success criteria, which helped us identify failures and decide what to change in the prototype.
