What Three Famous Metrics Really Measured

Goodhart’s Law says that once a measure becomes a target, it stops being a good measure.

Airbnb: nights booked

 

A night gets booked only when a guest wants a place and a host has one, so the metric is a proxy for both sides of the marketplace working. Counting nights rather than bookings ties it to revenue, since longer stays earn larger fees. It records the purchase rather than the stay, so a team that hid fees until checkout would raise it while misled guests left.

Facebook: 7 friends in 10 days

Chamath Palihapitiya’s growth team found new users who added seven friends within ten days usually stayed. Seven friends is roughly enough to fill a feed with new posts from people the user knows, and ten days tests whether that happens before interest fades. The metric counts connections rather than relationships, so a team could hit it by suggesting loose acquaintances and fill feeds with posts users ignore. The link is also correlational, since users with friends already on Facebook would likely have stayed anyway.

Slack: 2,000 messages

Slack found that 93 percent of teams that sent 2,000 messages kept using Slack. By then a team has moved its conversations out of email and built a searchable history it would lose by leaving, so the team depends on Slack. Because the metric counts volume, a team could reach it with bot messages, and the count would rise while people muted noisy channels.

All three were clever because each tracked a sign that users found value. All three were weak because they counted activity, which a team can produce without value once the count becomes a target. I would pair each with a guardrail: repeat guests, interactions with new friends, and replies between people.

Avatar

About the author

Leave a Reply