By Jaryd Hermann, founder, product manager and writer. Recipient of The Mail&Guardian, 200 Young South Africans in Business and Technology.
In Eric Ries’s famous book, The Lean Startup, he presented the notion of lean product development and validated learning. This is now a widely popular framework of “build, measure, learn” — a continuous loop to build and ship the least amount of product as quickly as possible. This also allows you to maximize your learning potential, validate your assumptions and hypotheses and iterate based on what you discover.
At the heart of this learning concept is an iterative product development process cycle called measurement. And for a founder or product manager, having access to informative, actionable, and reliable data and analytics is essential to having visibility and insight into how your product is performing and being used by people.
With the above knowledge, a common mistake is to track and measure everything in tools like Amplitude. This leads to dashboards and reports showing granular clicks and actions that can easily be misleading and distracting. And ultimately, this opens you up to analysis paralysis, where it becomes increasingly harder to see what actually matters and take appropriate action.
A few well-chosen events and properties are far more valuable than myriad data points.
Asking the right questions is at the core of measurement. Here are four approaches to soliciting effective and targeted questions so you and your team can better focus on what data to measure.
Decide why you need to collect data.
The foundation of choosing what to measure is knowing what you need insight into and what decisions you are hoping to inform. This is a critical part of collecting and maintaining good data.
For example, do you need data to inform a decision, such as whether an opportunity will have a revenue-generating impact on the business? Or perhaps reduce uncertainty around an assumption?
Another common need is understanding the performance and impact a certain product or feature has. It’s important to know if something is or is not working.
Brainstorm different questions.
Once you know your data needs, you’re in good shape to start brainstorming questions to inform your analytics. The importance of having these questions fleshed out cannot be overstated, and it’s vital to have input from your team, stakeholders and cross-functional teams.
For example, the product team will want to have insight into a different set of variables as compared to marketing or operations. With your team, start a conversation and brainstorm the different questions that map to each need. These questions should range from broad-based questions to more specific and granular questions:
• Decisions: Should we even build X?
• Assumptions: The most important thing for X type of person is Y.
• Performance And Impact: Is this new feature impacting subscription retention?
Once you have these decisions, assumptions, and performance questions written down, start exploring them further by breaking them down into further detailed who, what, why, where, when and how questions.
Make space for ‘bad’ questions.
As you probably learned in school, there is no such thing as a bad question. And as a founder or product manager, it’s important to create a safe space where any question goes. Often the less formed and more surface-level questions inspire better teams to develop even better questions. Finding and settling on the high-impact questions as a team is an iterative process, and individuals need to feel comfortable enough to participate in that process, even if they don’t have a stellar answer right out of the gate.
Prioritize your questions for long-tail insights.
At this point, you’ll probably have ample questions related to each data-need, and it might be enticing to have data reporting and analytics in place to answer all of them. However, you want to measure what’s most valuable and actionable. From your question set, prioritize based on where it’s most valuable to learn, and what types of questions you believe will unlock longer-tail insights.
Remember, instrumenting analytics to measure everything is not the right approach. It takes engineering hours to set up, and risks hours of getting caught up in the streams of data you have on your dashboards. There is not always a linear relationship between what is instrumented and the insights you can unlock.
Focus on asking the right questions that have a meaningful impact on your product, and strategically instrument a handful of events and charts that can enable a wide range of insights.