Webinar: The Skeptic's Guide to Data by Deliveroo Sr PM

Summary

スクリプトの要点を簡潔にまとめ、ユーザーの興味を引き出す魅力的な概要を提供する。

Takeaways

• 🗣️ データは製品マネージャーが現実世界と製品がどのように相互作用するかを理解するために必要な真実の反映として扱われています。
• 🤔 データは単なるツールであり、適切に使用され、目的に応じて扱われなければなりません。
• 💡 データの能力と限界を理解し、他のツールと組み合わせて使用することが重要です。
• 📈 データはオペラ指揮者のように、オーケストラのさまざまなセクションを統合して豊かな音の景色を作り出す必要があります。
• ❓ データの収集、処理、提示の動機を理解することが、データの最大限の活用と正しい意思決定を確保するための鍵です。
• 🎯 正しいメトリックスを測定しているかどうかを確認し、問題を理解し、影響を測定するために必要なものであることを確認します。
• 🚫 データが常に正しいわけではなく、誤解されることがあるため、常に批判的であり、自分の前提を疑問にすることが大切です。
• 🔍 データの生成方法を理解し、その正確性と信頼性を確認することが、データの品質を向上させるために不可欠です。
• 🤝 ユーザー研究とデータ分析は互いに補完し、製品発見のための聖三一です。
• 🛫 データとユーザー洞察を組み合わせることで、美しくて実用的なものを作成することができます。
• 🥳 データに愛を注ぎ、それを批判的に検証し、理解し、全力で活用することで、素晴らしい製品を作り上げることができます。

Q & A

W Perkinsが話すデータ分析の目的は何ですか？

W Perkinsは、データ分析が製品マネージャーが現実世界と製品がそれとの相互作用を理解するために必要な、信頼性の高い理解を得るための重要なツールであると述べています。

トニー・ベンの5つの質問とは何ですか？

トニー・ベンの5つの質問は、「何の力を持っているのですか」「その力はどこから得たのですか」「その力を誰の利益のために使うのですか」「誰に対して責任を負うのですか」「どうやって私たちから取り除くのですか」という連続的な質問です。

データが「嘘をつかない」という考えに対して、W Perkinsはどのような立場を取ていますか？

W Perkinsは、データが「嘘をつかない」との考えに対して批判的な視点を持ち、データは誤解されることがあるため、製品マネージャーはその情報を扱い的时候就には批判的な考えを持っていることが重要だと述べています。

データの収集、処理、提示の動機を理解するために、W Perkinsはどのようなアプローチを提案していますか？

W Perkinsは、データの収集、処理、提示の動機を理解するために、データが提示される背景や目的、そしてデータがどのように生成されたかを掘り下げることが重要だと提案しています。

W Perkinsが指摘する、「正しいメトリックス」とは何を意味しますか？

「正しいメトリックス」とは、製品マネージャーが問題を理解し、影響を測定するために必要な、適切なデータ指標を意味しています。重要なのは、データが実際に必要な情報を正確に反映しているかどうかを確認することです。

データの生成方法を理解することは、なぜ重要ですか？

データの生成方法を理解することは、データの完全性や信頼性を評価し、データが意図したメッセージを正確に伝えているかどうかを確認するために重要です。また、ユーザーの動機や製品との相互作用を理解することも重要です。

W Perkinsはデータの検証に関してどのようなアドバイスをしていますか？

W Perkinsは、データの検証においてユーザー洞察を利用することが重要であると述べています。データだけを頼りにせず、ユーザー研究や顧客洞察を組み合わせることで、より正確な理解を得ることができます。

データとユーザー洞察の違いは何ですか？

データは製品のユーザーとの相互作用を記録し、詳細が豊富な技術的なドローイングに似ていますが、ユーザー洞察はより印象主義的な絵画に似ており、世界に対する鮮やかなイメージを提供します。しかし、データは構造的な表現であり、ユーザー洞察は直接的な測定が困難です。

W Perkinsは製品マネジメントにおいて、どのようにデータを活用することを提案していますか？

W Perkinsは、製品マネジメントにおいてデータを愛し、批判的に取り扱い、理解し、完全な潜在力を発揮させることを提案しています。また、データとユーザー洞察を組み合わせることで、素晴らしい製品を構築し、製品マネジメントの良いニュースを広めることも重要だと述べています。

W Perkinsが提案するデータ分析の5つの質問は何ですか？

W Perkinsが提案するデータ分析の5つの質問は、「データセットの収集、処理、提示の動機は何ですか」「私が見るメトリックスは私が気にしているものであり、正しいことを測定しているのですか」「データはどのように生成されており、高い完全性を持っていますか」「どのようにしてそのデータを検証することができますか」「データを使用して意思決定をサポートする人たちをどのように参加させますか」です。

W Perkinsはなぜ、製品マネージャーには批判的な思考が必要だと述べていますか？

W Perkinsは、製品マネージャーは最終的に意思決定について責任を持つため、情報を扱う的时候就は批判的な思考が必要です。データが誤解されることがあるため、製品マネージャーは決断を下す前に、その情報を厳密に検証する必要があります。

Outlines

00:00

🗣️ データの効果的な使用と製品マネージャーの役割

この段落では、スピーカーであるW Perkinsが、データの効果的な使用方法について語ります。特に、製品マネージャーがデータをどのように扱うか、そしてデータが製品との相互作用を理解する上で役立つことについて説明します。また、政治家Tony Bennの5つの質問を引用して、製品マネージャーがデータを使って真実を探るべきだと主張しています。

05:04

🤔 データのモチベーションと正しいメトリックスの重要性

この段落では、データが製品マネージャーにとって重要な決定支援ツールである一方で、データがどのように生成されるか、そしてそのモチベーションを理解することがどうやって重要なのかについて説明します。また、データが提示された方法を理解することで、正しいメトリックスを見出すことができ、データの生成方法を理解することで、データの信頼性を高めることができます。

10:06

🛡️ データの生成方法と検証の重要性

この段落では、データが人間によって生成されること、そしてデータの完全性と信頼性について議論します。データが製品との相互作用を理解するための重要なツールである一方で、データの生成方法を理解することが、データの信頼性を評価する上で不可欠です。また、データの検証方法についても触れ、データが正しく解釈されるかどうかを確認する重要性について説明します。

15:08

🚀 データとユーザー洞察の相補性

最後の段落では、データとユーザー洞察の相補性について語ります。データは詳細で正確な情報が含まれる技術的な図面に例えられ、一方ユーザー洞察は印象派の絵画に似ており、生命あるイメージを提供します。これらの2つを組み合わせることで、美しいだけでなく実用的なものを作成することができます。製品マネージャーはデータとユーザー研究をバランス良く利用し、製品をより良いものにするためのヒントを提供します。

Keywords

💡データ

データは、製品マネージャーが現実世界と製品がどのように相互作用しているかを理解するために使用するツールです。このビデオでは、データが製品マネージャーのオーケストラの第一小提琴手として機能するが、他のツールと組み合わせて使用される必要性についても述べています。データは、製品マネージャーが正しい意思決定を行うために信頼性の高い理解を提供する反射像として機能します。

💡製品マネージャー

製品マネージャーは、製品の設計、開発、および成功に必要な戦略を策定する責任を負う人物です。このビデオでは、製品マネージャーがデータを使って意思決定をすることを強調し、データの正しい理解と使用の重要性について説明しています。

💡意思決定

意思決定とは、製品マネージャーが製品の開発やマーケティングなどの戦略を立てる際に行われるプロセスです。このビデオでは、製品マネージャーがデータを使って正確な意思決定を行う方法について説明しています。

💡データの整合性

データの整合性とは、データが信頼できる、正確で一貫性があることを指します。このビデオでは、製品マネージャーがデータの整合性を理解し、データに基づく意思決定を行う重要性について強調しています。

💡データの検証

データの検証は、データが正しい解釈を提供していることを確認するプロセスです。このビデオでは、製品マネージャーがデータを使って意思決定を下す際に、そのデータが正しい情報を伝えているかどうかを検証する必要性について述べています。

💡ユーザー洞察

ユーザー洞察とは、ユーザーのニーズ、欲求、行動に関する深い理解を指します。このビデオでは、データ分析と組み合わせてユーザー洞察を使用することで、製品マネージャーがより効果的な意思決定を行うことができると強調されています。

💡意思決定の責任

意思決定の責任とは、製品マネージャーがその職務を行う際に、その意思決定が製品やサービスに与える影響に対して責任を持つことを指します。このビデオでは、データを使って意思決定をする際に、製品マネージャーが自らの責任を認識し、データの正確性や信頼性について慎重に取り組む必要性が強調されています。

💡データの偏見

データの偏見とは、データが完全な真実を反映していない場合や、解釈の余地がある場合に発生するバイアスや誤解を指します。このビデオでは、製品マネージャーがデータに偏見がある可能性があることを認識し、常に批判的思考を行うことが必要です。

💡モチベーション

モチベーションとは、データが収集、処理、提示される背景にある理由や意図を指します。このビデオでは、製品マネージャーがデータのモチベーションを理解することが、データの正しい解釈と使用に不可欠であると述べられています。

💡製品設計

製品設計とは、製品の外見、機能、使いやすさなどを考慮して、製品を創造するプロセスを指します。このビデオでは、製品マネージャーが製品設計においてデータを使って効果的な意思決定を行う方法について説明しています。

💡問題認識

問題認識とは、製品マネージャーが市場や顧客のニーズを理解し、製品開発に必要な問題点を特定するプロセスを指します。このビデオでは、データを使って問題認識を行うことの重要性について説明されています。

💡製品戦略

製品戦略とは、製品マネージャーが製品の将来の方向性や目標を定義し、その実現に向けて計画を立てることを指します。このビデオでは、データを使って製品戦略を策定する方法について説明しています。

Highlights

W Perkins, a product manager at Deliveroo, emphasizes the importance of effectively using data in product management.

Drawing inspiration from Tony Benn's five questions, Perkins suggests adapting the framework for data analysis in product management.

Product managers must understand data's abilities and limitations, using it as one of many tools in their decision-making process.

Perkins highlights the need for skepticism in data interpretation, as data can be misinterpreted or used to lie.

The 'data doesn't lie' adage is critiqued, with Perkins urging product managers to question assumptions and avoid naivety.

Perkins discusses the motivation behind data collection and presentation, using advertising as an example of data manipulation.

The importance of choosing the right metrics is stressed, illustrated with the historical example of the steel helmet in World War I.

Perkins warns against relying solely on data, advocating for a balance between quantitative and qualitative insights.

The concept of data integrity and its impact on decision-making is explored, emphasizing the human element in data creation.

Perkins advises product managers to understand user motivations when gauging the integrity of data sets.

The speaker shares insights on how to monetize user behavior, using Tinder's premium feature as an example.

Validation of data through user research and insights is crucial, as highlighted by Perkins.

Perkins describes the synergy between user research and data analysis, calling them the 'Holy Trinity' of product discovery.

The importance of fostering a community and encouraging aspiring product managers is discussed.

Perkins concludes with a call to love data, but to remain skeptical, validate, and understand data to build great products.

Transcripts

hello ladies and gentlemen product

managers and product enthusiasts all my

name is W Perkins and I'm a product

manager at deliveroo I'm delighted to be

here with you today alas I speak to you

over the airwaves but I hope one day to

meet you in person at a product School

event or something similar I'm truly

humbled to be able to address you today

and steal I hope just 20 minutes of your

time to talk to you a little bit about

how I think about effectively using data

particularly when it comes to you in a

form where you may not necessarily be

familiar with it and is perhaps

presented to you by stakeholders or a

third party so without further Ado let

me take you back to 1970s Britain where

the radical Fringe of leftwing political

thinking was dominated by the labor

Firebrand MP and pipe smoking

afficionado Tony Ben skeptical of those

in power he formulated his famous five

questions to ask those with it in order

to get to the truth about how they

intended to wield that power what power

have you got where did you get it from

in whose interest do you use it to whom

are you accountable and how do we get

rid of

you this he believed would tear away the

veils of obscuration and get down to the

truth of the

matter now as product managers we are

also interested in the truth in order to

make good well-reasoned rational

decisions we need to have a reliable

understanding of what is going on in the

real world and how our products interact

with it an excellent and some might say

the best reflection of that real world

is data but data is merely a tool and

like all tools must be wielded

skillfully and to a purpose we must

understand its abilities and

limitations it's also not a Swiss army

knife to achieve our goals it needs to

be used in conjunction with other tools

in their toolbox so like an operatic

conductor product managers must pull

together the various sections of the

orchestra to create that rich soundscape

and create that picture data may be

playing first violin in our Orchestra

but it needs the support of the others

to really hit those virtuo high

notes so as Tony Ben interrogated

political power to find truth so can we

with our

data why don't we take Ben's five

question framework for divining truth

and create our own that we can use to

make the most out of our data and ensure

we are making the right decisions for

our

product I want to know what motivates

the collection processing and

presentation of a data set I want to

know whether the metrics I'm looking at

are the are actually the ones I care

about and are the right thing to be

measuring I want to know how the data's

been generated is it high or low

integrity and once I'm happy with that I

want to know how I can validate that

data ask whether it's really telling me

what I think it is and finally I must

say I'm not quite as much of a

revolutionary as old Tony so I'll take

his final question and turn it on its

head if somebody's interested in using

data to inform their decision-making

then how do we get them on board as

product

managers now a me sure you often hear is

that the data doesn't lie but think

about that for a minute think about the

data you asking and buy on a daily basis

outside of your

work generally I find it in advertising

either trying to sell you a product or a

political idea perhaps in this context

one could be forgiven for thinking it

does nothing but

lie the data doesn't lie is a nice

sentiment and I get it you know it's

inanimate and human interactions

required before it means anything but

frankly to take this phrase at face

value is a little

naive and the one thing that product

managers cannot be is naive we've got to

be sharp we've got to understand what

we're talking about and so we've got to

be skeptical about any information we're

using to make those decisions because

ultimately we and we alone are

accountable for those decisions no one

else and we certainly cannot blame the

data we're

using data it can be wrong it can be

misinterpreted and yes on occasion it

can be used to lie now please don't see

this as cynical or dismissive of it I

think skepticism is healthy questioning

our assumptions will help us to better

understand what is true and what isn't

it will help us avoid blunder improve

the quality of our decision-making and

ultimately will make us better product

managers so then let's get stuck in with

question

one what is the

motivation let's pick up that thread

again of where we see data in our

day-to-day lives outside the world of

product management if such a thing does

exist we mentioned that people use data

to try and sell us things and often we

won't even notice or recognize that data

is being

used now apparently it's one of those

rules of the internet that um one must

have cats in their presentation now I

don't usually go in these sort of things

but rules is rules so here we

are I don't know if any of you remember

this classic I guess 880s or 90s um

whiskers advert campaign for cat food

eight out of 10 owners who expressed a

preference said their cat preferred

whiskers I know it's a bit of a mouthful

isn't it but um this a has some fairly

compelling data in it so it must be true

and therefore we should all go and buy

this

product well I don't know if you're

anything like me but when I see things

like this I had to go have a bit of a

lie down in a dark room because my my

product manager brain just goes off the

wall with it a little bit you know what

does this mean you asked 10 people and

eight of them said said so or 80% of

10,000 people said it and who were these

people do they often present their cat

with a a selection of dishes for their

supper and then what do they prefer it

to did they prefer it to a a succulent

roast chicken or perhaps going

hungry what about the owners who didn't

Express a preference what do they

think in this case I'm

skeptical this is obviously about

selling as a product and we have no way

of knowing whether it's true or not and

yet and yet there's a bit of me that

still just nods along and says okay next

time the cat's hungry I'll go and buy

the whiskers you know it's effective it

Taps into our psyche somehow and

advertisers wouldn't do it

otherwise we face this type of thing as

product managers in our jobs people will

regularly approach us with ideas for

things they want us to build and a great

way to sell that idea is supporting it

with data which frankly is brilliant far

better than doing without and we should

wholeheartedly encourage

it but it's totally natural for them to

want to make that data as compelling as

possible it's their

job but it's also our job as product

managers to understand why they are

showing us this data and unlike in an

advert we have the opportunity to put

our skepticism into

practice we can question what their

motivations are for presenting their

data through this we can understand why

that data has been presented in the way

it has what is present in it it what is

missing what assumptions have been made

you we could even do another whole video

on this a subsection of 20 questions to

ask when you thinking about the

motivation for things but that's for

another time and and another

place this is also our first opportunity

to understand whether we looking at the

right data and the right metrics both in

order to understand the problem um and

also to measure the impact which lead

provides with quite a nice segue into

question

two are these the right

metrics so join me as we venture further

back in time from the 1970s to

1914 World War I and Europe is a flame

as the major European powers confront

theel confront each other on the fields

of northern France in the decades since

the last major European War military

technology had taken great strides

forwards but the tactics in use will be

recognized by a soldier fighting in

those same Fields a century earlier at

the Battle of watero officers wore white

gloves and ostrich plumes in their hats

whilst leading troops the mounted

cavalry charge was still a common sight

in the

battlefields soldiers of all Nations

went into battle wearing cloth headwear

even the rather satisfyingly named

German pickle halber with its

distinctive brass Spike and Crest on the

front was just made of a rather thin

leather provid little bistic

protection as the machine gun made

surface Warfare impossible trenches were

dug deep into the ground to provide

cover artillery artillery bombardment

then became the modest operandi for all

armies now when artillery shell bursts

it would blow up into the air huge

amounts of Earth rubble and rocks that

would then rain down on those Sheltering

in the

trenches military commanders noticed

that this was resulting a number of

soldiers being taken back to Aid

stations with head injuries increasing

rather

dramatically now British commands in

particular picked up on this as a good

metric to try and address field Marshall

the old hay then will play the role of

our product manager

here he took measures to reduce the

number of head injuries sustained by his

troops by introducing the steel helmet

smartly modeled by the chap pair on the

left um The Mark 1 bro helmet that's

called this is a very good example of

product design it's protective has a

nice rim to Shield the eye to Shield The

Shield the eyes from the Sun and on the

back of the neck from anything falling

behind you it's also got a sort of

rather jawy aesthetic which soldiers of

all of all times rather appreciate

but interestingly and rather alarmingly

once these helmets were introduced the

rate of head digies actually increased

and in and increased dramatically now

why was this one theory was that the

added feeling of safety the helmet

provides led to soldiers acting more

recklessly taking more risks sticking

their head above the parapet where

before they might not

have yes I suppose this is a viable

hypothesis but it's based on an

incomplete understanding of the data set

head injuries per thousand is not the

only metric that is important here the

most important metric particularly for

those actually having to wear the

helmets is the rate of death yes the

head wounds increased but only because

soldiers who would otherwise have been

killed by shrapnel were instead only

sustaining

injuries so here our product manager

found a good metric that got him onto

the target but he needed to go one level

deeper in his understanding to get to

where he actually needed to be

I often think about in terms of um

prospecting for gold you find traces of

the precious metal on the surface and in

streams but one needs to dig deep

underground to find that main gold

them so we think we have our metrics we

want to measure and move well before we

get too excited we need to think about

how reliable this data is and one way to

do that is think about how it's been

generated we must remember that data

doesn't emerge out of The Ether pristin

and Incorruptible like I can think of it

like the body of one of those preserved

Saints in a Venetian church that said

sort of waft the scent of flowers even

after centuries in the Tomb instead we

must recognize that at some point

somewhere somebody has written a bit of

code that creates his data based on a

set of assumptions valid at the time and

to the best of their knowledge available

knowledge which may have since developed

and

changed thinking of this data think of

data in this way I find helpful

it is the work of human hands and and as

such is susceptible to all the failings

of human endeavor continuous iteration

of development can improve the Integrity

of data but it can also result in errors

and misunderstandings getting

institutionalized we should be mindful

of this and if we if we aren't familiar

with the data we should check it

honestly I cannot tell you the number of

times I've been confused by naming

conventions thinking that one metric was

one thing when in fact it was measuring

something complet completely

different now data is gener generally

created by your users's interaction with

a product when someone presses a button

on your app or on your website it findes

an event and then builds up a data set

so again you need to understand the

motivations of your users when they're

engaging with your product in order to

gauge the Integrity of that data

set let's look at two examples web

traffic is Fairly reliable you know when

someone has landed on your web page and

there generally list room for error or

motivation for misuse but even here you

may find a deliberate denial of service

attack on your website can make it look

like your size is extremely popular when

in fact it's a Mali actor trying to

bring it

down and at the other end of the

spectrum a common uh source of data for

uspm comes from customer support now I

think this is generally an invaluable

way of understanding the pain points for

our products but just remember that

often this this data is generated

through the manual tagging of issues by

agents who have to interpret the reasons

for the

contact generally this is fine but

sometimes it can create distortions as

agents try to squeeze issues that aren't

catered For into the existing

categories my top tip here is that if

your most common category is other then

you need to be very very careful with

how you use that

data also if you have anything gamified

in your product you've got to be careful

somebody very well worldly wise once

said to me if you create a game then

players are going to play it and people

play to win even if that means cheating

users will do what they need to do in

order to achieve their

goals I think a great example of this

and I think a truly awesome way of

addressing it is through the dating app

Tinder where one arranges romantic

Liaisons with other like-minded people

in in your their local area now Tinder

saw that people didn't necessarily want

to be restricted to their local area

area they wanted to search far and wide

for their sweetheart and so people began

to mock their location data in order to

check out potential dates in other

areas here users were purpose

purposefully creating Incorrect and

misleading data in order to achieve

their goals now if anyone's ever engaged

with this sort of thing location mocking

is pretty difficult to stop and so

rather than trying to to stop this um

outright I think whoever the PM was at

the time um had a brilliant idea to not

just embrace it but make it a premium

feature giving people the ability who

pay for it um the ability to search

anywhere they want

it it was understanding how the data was

generated recognizing the anomalies in

it and pairing that with a knowledge of

the motivations of the user that they

found a really really Innovative way to

monetize their

product so on our next question then how

can it be

validated

for those non-plan spotters and people

listening on audio we're looking at the

cockpit of a 787 Dreamliner one of the

most advanced aircraft in the world now

it's very very possible to fly This

Plane using the data from purely the

instruments and in fact much of the time

the aircraft can fly itself on nothing

but the

data but not everyone is flying a

Dreamliner most of us product managers

are working startups or scale UPS where

the cockpit perhaps looks something more

akin to

this now one could possibly fly this

just by looking at the dials with a map

and compass on your lap but I think it's

a good idea here to actually take a look

out the window and see what's going on

around you to perhaps push this analogy

One Step Beyond its limit a lot of the

time in product development we don't

even get the the plane to start flying

we have to throw ourselves off the cliff

and build the plane on the way down this

includes conceptualizing and discovering

the metrics that we're going to use

before we even build the instrumentation

to measure them sometimes we have no

idea what we should be measuring and

there are infinite number of things that

we could measure out

there I think it's here that user

research or customer insights or just

your knowledge of the user really comes

into

play where you don't know what you

should be measuring user research can

provide you those insights and tell you

what is important if you find yourself

lacking data it can provide you with a

good read on what you should do

next even if you're flying that

Dreamliner it's a good idea to have a

look at the window every now and again

and just validate the data make sure

it's really telling you what you think

it is so always look out the window ask

yourself how can you validate data with

user insights and on the flip side how

can you quantify user insights with

data now if you're perhaps indulging for

a moment longer on this subject I might

take us a bit further I've spoken a bit

about how data is a reflection of the

real world but this is really sloppy use

of my language because data is not a

reflection it's not a true if reversed

image of the

world it's more of a technical drawing

that can have an incredible amount of

detail but lacks the color and and life

of the real

world counter to this though user

insights is more in the style of an

impressionist painting a mon or says

creating a vivid image of the world but

again it's very difficult to measure

anything on one of these paintings one

wouldn't necessarily want to build a

structure from them but nor any will

want to live in a

schematic the two complement each other

very well because they opposite sides of

the same coin and when you combine them

both you can create a thing that is

beautiful but also

practical I see products user research

and data analysis as the Holy Trinity or

for the more ancient Roman minded

obvious the triumverate of product

Discovery everything you need is in that

Golden Triangle my my top tip here is to

try and sit between your user researcher

and your product analyst and keep up a

conversation between you I think that

the insights you gain from that

conversation will will really drive your

product to to to new

heights and don't just take my word for

it this guy used to work at Amazon and

he says a similar thing he says the

thing I've noticed is when the anecdotes

and the data disagree the anecdotes are

usually right there's something wrong

with the way you are measuring it now

big Jeff isn't saying dismiss the data

for anecdote he's literally saying be

skeptical if your gut is telling you

something isn't right and as a PM you'll

develop a great Instinct for these

things then it probably isn't and you

should check

it and so finally where Ben wants to

kick you out in the product management

World we're a bit more accommodating and

friendly and we're going to ask ask how

do we get you on board I just think if

somebody has shown the interest and the

spark to take identify a problem take a

bit of data put it around it maybe

perhaps come up to hypotheses we want to

think about how we can get them into the

community and encourage them to become

product managers themselves I think

organizations like the product school do

a brilliant job in trading out people

who want to be PMS but I know that when

I was coming into the industry I found

the first hurdle was even knowing what

PM was and it was a and it was a thing

that I could

do so I would encourage you to take

these people under your wing teach them

about product management and product

thinking and if nothing else you'll have

a group of people around you in your

business who you can rely on to provide

you with excellent material for your

product

roadmap so then to finish love your data

but be skeptical of it validate it

understand it use it to its full

potential and you'll build awesome

products and finally get out out there

and spread the good news of product

management thank you very

much