PostHog: who gets the new feature, and what should you watch?
PostHog combines product analytics, session replay, experiments and feature flags. Teams can open a new feature to selected users, then use events, replays and errors to decide whether to expand it.
Form your own approach first, then explore the reasoning to check trade-offs and validation you may have missed. There is no single right answer.
What must an expansion confirmation explain?
Scenario
Design a confirmation panel for expanding a new form on your booking website. The new form currently serves selected logged-in users. Operations wants to move from 10% to 30%, but a higher configured percentage does not prove better results. This is an invented design exercise, not a recreation of PostHog.
Known conditions
- A fixed eligible population contains 1,000 valid user IDs allocated through stable buckets. Currently 92 actually match; the 30% preview matches 284. Both previews use the same identities, rule version and eligible population. Actual counts need not equal the nominal percentages.
- Of the current 92, only 40 started the form and 32 submitted successfully. The other 52 did not start and must not be counted as form failures. The old form's concurrent results come from a different population; raw success rates cannot establish a causal win.
- Recent replays show one definite date-validation error and two unexplained anomalies. The team may expand, hold or pause the new path, but each decision needs a rationale and someone responsible for subsequent observation.
- After the preview is generated, a colleague changes the rules. The page must require fresh confirmation rather than submit the old preview. Design visible states and actions; do not implement the release backend.
Deliverables and walkthrough
Sketch the confirmation panel and stale-preview state. Explain what 92, 284, 192, 40 and 32 mean. Give one reason to hold expansion and identify evidence that would allow reconsideration, without assuming a single correct release decision. Design two observation tasks to check understanding of affected people and sample boundaries. Do not invent test results.
Explore the reasoning
- Explain who changes experience. With the same fixed population and buckets, 284−92=192 users are newly included; the original 92 remain included. Separate the percentage setting, actual matches and new matches. Do not turn 30% into exactly 300 users.
- Separate the usage funnel. There are 92 matching users, 40 starters and 32 completions. The completion rate among starters is 32/40; the 52 non-starters are not automatically failures. Raw rates from different populations do not prove the new form caused an improvement.
- Keep unresolved problems visible. Show the date error, two unexplained anomalies and their evidence. Expansion depends on risk, observation time and critical tasks, rather than one green rate.
- Bind confirmation to the preview version. Changes to the population or rules make a preview stale. Preview again and reconfirm. Retain the old preview to explain the previous judgment, not to authorize a new configuration.
- Test understanding rather than click speed. Ask users how many people will be added and what the denominator in 32/40 represents. Then show a stale preview and observe whether they recheck it. Use misunderstandings to improve information hierarchy.