Webinar How to Measure Product Success by Squarespace Sr PM

Summary

In this insightful session, the speaker, a product manager at Squarespace, discusses the importance of measuring product success accurately. They caution against relying solely on vanity metrics, which may appear impressive but lack depth, and emphasize the need to add context like time frames and conversion rates for meaningful insights. The presentation also highlights the potential pitfalls of percentages, which can be misleading without supporting metrics, and underscores the significance of time series analysis to discern trends and seasonality effects. This comprehensive approach ensures product managers can make informed decisions and develop effective strategies.

Takeaways

• 📊 Understanding Product Performance: It's crucial for product managers to have a clear understanding of product performance and the metrics they are measuring to inform their strategy.
• 📌 Defining Right Metrics: The session aims to help avoid pitfalls in defining the right metrics to measure product success effectively.
• 🚫 Avoiding Vanity Metrics: Product managers should look beyond vanity metrics that may look impressive but don't provide actionable insights into product performance.
• 🔍 Contextualizing Metrics: Adding context such as time frames, data traffic sources, conversion rates, and user engagement is essential to make vanity metrics actionable.
• 💡 Vanity Metrics Utility: Vanity metrics can be useful as benchmarks for early-stage products or new features to set milestones for success.
• ⚠️ Misleading Percentages: Percentages can be misleading; it's important to have secondary metrics to provide a full picture of product performance.
• 📈 Time Series Analysis: Time series data is important to understand seasonality effects and long-term trends, which can give a more accurate view of product success.
• 📉 Seasonality Impact: Recognizing and accounting for seasonality is vital to accurately assess the impact of new features or changes in product strategy.
• 📚 Importance of Absolute Numbers: In addition to percentages, absolute numbers provide a concrete measure of performance, helping to avoid misinterpretation of improvements.
• 🕵️‍♂️ Product Sense: Product managers should use their judgment to interpret metrics in context, making informed decisions that consider both quantitative and qualitative data.
• 🗣️ Communication of Metrics: Effectively communicating the meaning and implications of metrics is key for product managers to set realistic expectations and strategies.

Q & A

What is the main focus of the session presented in the transcript?

The session focuses on how to effectively measure product success as product managers, emphasizing the importance of understanding product performance metrics and avoiding pitfalls in defining the right metrics.

Who is the presenter of the session?

The presenter is a product manager at Squarespace, part of the growth team, responsible for conversion strategies and user trials.

What is the presenter's background before joining Squarespace?

Prior to Squarespace, the presenter worked as a product manager at ONC, a small business lending company, where they had the opportunity to work on different products and customer journeys.

What is the presenter's educational background?

The presenter has an engineering background and leveraged a business degree to transition into product management.

What are 'vanity metrics' according to the session?

Vanity metrics are those that may look good or impressive on the surface but do not provide any understanding or context into what is happening, such as the number of app downloads or website visitors.

Why are vanity metrics considered misleading for defining future strategy?

Vanity metrics can be misleading because they do not help in understanding the underlying reasons behind the numbers or inform future strategy, as they lack context and actionable insights.

How can vanity metrics be made actionable?

Vanity metrics can be made actionable by adding context such as time frames, data traffic sources, conversion rates, and per user or visit rates, which can help in understanding the story behind the numbers.

Why might companies still use vanity metrics even though they can be misleading?

Companies might use vanity metrics as they can be helpful in setting benchmarks or milestones for success, especially during the early stages of a product launch or feature release.

What is the issue with relying solely on percentages to measure improvement?

Percentages can be misleading as they may not provide the full picture of the situation. A higher or lower percentage might not accurately represent the actual improvement or decline without considering the base numbers.

How can secondary metrics help in providing a more accurate picture of improvement?

Secondary metrics, such as absolute numbers of landings or relative percentage changes, can provide additional context and help in understanding the real impact of changes, avoiding overestimation or underestimation of improvements.

Why are time series important in understanding product performance?

Time series are important because they help in identifying seasonality effects, trends, and the overall growth or decline of a product over time, providing a more comprehensive understanding of the product's performance.

What is an example of how time series can affect the interpretation of product performance?

An example given is a business selling plants, where a sudden increase in sales might be attributed to a new feature. However, looking at the full year's time series could reveal that the increase is due to seasonality, and the new feature's impact might be less significant than initially thought.