Google PM Interview: Google Maps Korea Launch

Exponent
1 Dec 202122:20

Summary

TLDRIn this mock product management interview, Hal, an engineering manager with previous experience at Google, discusses strategies for Google Maps' potential launch in South Korea. He evaluates factors such as data quality, cost, and integration challenges when choosing between local vendors for map datasets. Hal emphasizes the importance of precision and recall in business search results and suggests A/B testing and user-generated content to refine data accuracy. The conversation highlights a user-centric approach to product management, focusing on the ultimate goal of providing the best user experience.

Takeaways

  • πŸ˜€ Hal, an engineering manager at Cressicore and former PM at Google, discusses choosing a map data vendor for Google Maps' launch in South Korea.
  • πŸ“ The decision-making process should consider data quality, cost, and non-functional aspects like integration and updates.
  • πŸ” Data quality is crucial for accuracy and user experience, including precision and recall of business listings and directions.
  • πŸ’° Cost may not be a primary concern for Google, which prioritizes the best user experience over vendor pricing.
  • πŸ”„ Non-functional considerations include the ease of integration, licensing, and the reliability of ongoing data updates.
  • πŸ“ˆ Hal suggests using A/B testing with user interaction metrics to evaluate different data sets' effectiveness.
  • πŸ“Š Metrics to monitor include interaction rates, search abandonments, and query refinements indicating user satisfaction.
  • 🏒 Google's infrastructure and user engagement features, like reporting incorrect data, can compensate for initial data set deficiencies.
  • πŸ”‘ The ultimate goal of Google Maps should guide the trade-off decisions between precision and recall of data.
  • πŸ€” Hal reflects on the importance of clearly defining the mission and goals at the outset and tying them back to decision-making later.
  • πŸ”‘ The interview concludes with feedback emphasizing the structured and use-case driven approach Hal took in addressing the problem.

Q & A

  • What is the main topic of the mock interview in the video transcript?

    -The main topic of the mock interview is about Google Maps wanting to launch in South Korea and the decision-making process for choosing a local vendor for map data sets.

  • Who is Hal and what is his background according to the transcript?

    -Hal is an engineering manager at a startup called CressiCore. He was previously a product manager (PM) at Google for five years, working on Google Assistant.

  • What are the three factors Hal considers when choosing a map data vendor for Google Maps in South Korea?

    -Hal considers data quality, the business aspect (including cost and contract terms), and non-functional aspects such as integration ease, licensing, and data reliability.

  • Why does Hal believe that data quality is the most important factor in choosing a vendor?

    -Hal believes data quality is the most important factor because it directly affects the user experience and Google's ability to provide the best service to its users.

  • What does Hal mean by 'precision and recall' in the context of map data quality?

    -In the context of map data quality, 'precision' refers to the accuracy of the data, ensuring that the places listed are indeed what the user is searching for. 'Recall' refers to the coverage of the data, ensuring that the data set includes all relevant places.

  • How does Hal suggest measuring the difference between the map data and reality?

    -Hal suggests using an A/B test with different data sets for a portion of users and comparing metrics like interaction rates, abandonments, and refinements. He also mentions the possibility of using human evaluators to gather 'golden data' for comparison.

  • What are some of the metrics Hal suggests monitoring during an A/B test of map data?

    -Hal suggests monitoring interaction rates, abandonments, and refinements as indicators of how relevant the search results are to the users.

  • How does Hal define the success metric for Google Maps in the context of the interview?

    -Hal suggests that the success metric for Google Maps could be whether users actually visit the places they find through the app, as it indicates the relevance and usefulness of the map data.

  • What is Hal's approach to dealing with trade-offs between precision and recall of map data?

    -Hal's approach is to understand Google Maps' ultimate goal or 'north star' metric and make a decision based on what aligns best with that goal. He also mentions the possibility of merging multiple data sets to balance precision and recall.

  • What feedback does the interviewer give to Hal at the end of the mock interview?

    -The interviewer appreciates Hal's use-case driven approach and his structured method of outlining considerations and diving deep into data quality. The feedback also includes the suggestion to refine the mission and goals at the beginning and to tie back decisions to the initial vision.

Outlines

00:00

πŸ—ΊοΈ Google Maps Expansion Strategy in South Korea

The video script begins with a hypothetical scenario where Google Maps is considering launching in South Korea and must choose between several local vendors for map data. The interviewee, Hal, introduces himself as an ex-Google PM and current engineering manager. The discussion focuses on the criteria for selecting a map data vendor, emphasizing the importance of data quality, cost, and integration feasibility. Hal suggests that for Google, data quality is paramount to ensure an excellent user experience, while cost and integration challenges are secondary concerns due to Google's resources and capabilities.

05:01

πŸ“Š Prioritizing Data Quality for Google Maps Launch

In this paragraph, Hal delves into the factors that determine data quality, particularly precision and recall of business listings. He explains that precision refers to the accuracy of the search results, ensuring that the places listed are indeed what the user is searching for. Recall, on the other hand, is about the comprehensiveness of the data, ensuring that all relevant businesses are included. Hal suggests that Google could potentially use A/B testing with different data sets to measure these metrics and make an informed decision on the vendor that provides the best data quality.

10:01

πŸ” Measuring Data Quality Through User Interaction

Hal discusses how to measure the effectiveness of map data through user interaction metrics. He suggests monitoring how users engage with search results, including interaction rates, abandonments, and refinements. These metrics can indicate the relevance and usefulness of the data provided. Furthermore, Hal proposes tracking whether users actually visit the places they find on Google Maps, suggesting that this could be a strong indicator of the data's real-world value. He also considers the possibility of using user-generated content to fill gaps in the initial data set.

15:02

πŸ€” Balancing Precision and Recall in Data Selection

The script addresses the trade-off between precision and recall in selecting map data. Hal acknowledges that one data set might excel in precision while another might have better recall. He suggests that the decision on which to prioritize depends on Google Maps' ultimate goal or 'north star' metric. Hal also contemplates combining multiple data sets or starting with one and using user contributions to improve it, highlighting the importance of identifying and addressing gaps in the data.

20:03

πŸ’‘ Reflecting on the Interview Process and Approach

In the final paragraph, Hal reflects on the mock interview process. He admits to jumping ahead at times and not clearly defining Google Maps' mission and goal at the outset. Hal believes that a more structured approach, starting with the mission and tying it back to decision-making later, would have been beneficial. The interviewer appreciates Hal's use-case driven strategy and his methodical dive into data quality, finding the interview valuable and indicating that Hal would be a good team member.

Mindmap

Keywords

πŸ’‘Product Management

Product Management is the process of guiding a product from its inception to its launch and beyond. In the video, the interviewee, Hal, discusses the strategic decisions involved in product management, such as choosing the right map data vendor for Google Maps' launch in South Korea. This concept is central to the video's theme of decision-making in a product management scenario.

πŸ’‘Google Maps

Google Maps is a widely used online mapping service developed by Google. The video's theme revolves around a hypothetical situation where Google Maps is considering launching in South Korea, and the interviewee must decide on the best local vendor for map data. Google Maps serves as the context for the discussion of product management strategies.

πŸ’‘Vendor

In the context of the video, a vendor refers to a local provider of map data for Google Maps' South Korea launch. The interviewee discusses evaluating different vendors based on factors like data quality, cost, and integration feasibility, which is a critical part of the decision-making process in product management.

πŸ’‘Data Quality

Data quality is a measure of how accurate, complete, and reliable the data is. In the video, Hal emphasizes the importance of data quality when choosing a vendor for Google Maps' map data in South Korea. It is a key factor because high-quality data leads to a better user experience.

πŸ’‘Precision and Recall

Precision and recall are metrics used to evaluate the quality of search results. Precision refers to the accuracy of the results, while recall measures the completeness. In the script, Hal discusses using these metrics to assess the data quality provided by different vendors for the businesses and places listed in Google Maps.

πŸ’‘A/B Testing

A/B testing is a method of comparing two versions (A and B) of a product or service to determine which performs better. In the video, Hal suggests using A/B testing to compare different map data sets from various vendors by giving different user groups access to each data set and measuring their interactions and satisfaction.

πŸ’‘Integration

Integration in this context refers to the process of combining the chosen map data with Google Maps' existing infrastructure. Hal discusses the ease of integration as a non-functional aspect to consider, including licensing and the ability to receive ongoing updates from the vendor.

πŸ’‘User Experience

User experience (UX) is the overall experience a user has while interacting with a product or service. The video's theme focuses on selecting the best map data to enhance the user experience on Google Maps in South Korea. Hal mentions that Google typically prioritizes providing the best experience to its users.

πŸ’‘North Star Metric

A North Star Metric is a single, overarching metric that aligns a company's efforts towards its ultimate goal. In the video, Hal suggests that understanding Google Maps' North Star Metric would be crucial in making the right trade-offs between different data quality aspects and vendor choices.

πŸ’‘Trade-offs

Trade-offs involve making decisions that involve balancing one desirable outcome against another. In the context of the video, Hal discusses the trade-offs between precision and recall in map data quality, as well as between urban and rural coverage, which must be considered when choosing a vendor.

πŸ’‘User-Generated Content

User-generated content refers to various forms of content, such as text, images, or data, that users create and share. Hal mentions leveraging user-generated content to fill gaps in the map data provided by vendors, such as reporting incorrect opening hours, which can enhance the overall data quality.

Highlights

Google Maps is considering launching in South Korea and evaluating local vendors for map data sets.

Interviewee Hal, an engineering manager with previous experience as a PM at Google, discusses the strategic decision-making process for choosing map data vendors.

Three key factors identified for decision-making: data quality, business terms (cost and contract), and integration feasibility.

Data quality is emphasized as the most critical factor due to its direct impact on user experience.

The importance of precision and recall in evaluating map data for business listings is discussed.

Coverage of map data, including its reflection of reality, is considered crucial for data quality.

Google's infrastructure and capabilities are highlighted as potential solutions for integration challenges.

The role of user-generated content in enhancing map data through reporting inaccuracies is mentioned.

A/B testing different data sets with user populations to measure effectiveness is proposed.

Interaction metrics, such as search refinements and abandonments, are suggested as indicators of data relevance.

The concept of 'golden data' collected manually to verify and ensure data accuracy is introduced.

The potential use of user engagement with map results as a proxy for measuring data value is discussed.

Trade-offs between precision and recall in map data are considered, with a focus on Google Maps' ultimate goal.

The idea of merging multiple data sets or complementing initial data with user-generated content is explored.

The interview concludes with feedback on Hal's structured approach and use-case driven strategy in evaluating data quality.

The importance of aligning decision-making with Google Maps' mission and vision is emphasized in the debrief.

The mock interview serves as a valuable example for viewers preparing for product management interviews.

Transcripts

play00:00

let's imagine that google maps wants to

play00:02

launch in south korea and they're you're

play00:05

choosing between several local vendors

play00:07

and each one has a different map data

play00:09

set tell me which one you would choose

play00:12

[Music]

play00:14

hey everyone welcome back to another

play00:16

exponent product management mock

play00:18

interview my name is kevin and on

play00:19

today's show we have how we're going to

play00:21

be doing a product management analytical

play00:24

or strategy type question but before we

play00:26

jump into that how can you just tell the

play00:28

audience a little bit about who you are

play00:30

hey everyone uh so i'm hal i'm currently

play00:32

an engineering manager in a startup

play00:34

called cressicore but before that i was

play00:36

a pm and 8pm at google for five years

play00:38

working on the google assistant

play00:40

sweet thanks for your time today so this

play00:42

is what i like to ask you let's imagine

play00:44

that google maps wants to launch in

play00:46

south korea and they're you're choosing

play00:49

between several local vendors and each

play00:51

one has a different map data set tell me

play00:54

which one you would choose

play00:56

okay so google maps wants to expand

play00:58

basically in a new country and i'm

play01:00

choosing between

play01:01

[Music]

play01:02

um different vendors for for maps data

play01:06

yes

play01:06

okay

play01:08

um

play01:09

and then how do i

play01:11

how do i decide basically

play01:13

okay so uh maybe just just to just to

play01:16

clarify but so when we say maps data i

play01:18

assume that's kind of like you know the

play01:19

data that i would see in you know the

play01:21

google maps app and uh so like when i

play01:24

when i open google maps in south korea

play01:26

like whatever i see there with the

play01:27

streets and the

play01:29

the businesses that that's what i'm

play01:30

that's what we're referring to here

play01:32

right

play01:32

yeah we can start with this assumption

play01:34

and then if we have time we can expand

play01:35

upon this definition cool okay yeah

play01:38

and

play01:38

um is this specifically like um like

play01:42

like you mentioned south korea is there

play01:43

a specific reason why why

play01:46

google maps is not in south korea yet or

play01:47

like

play01:48

why why are we why why are we expanding

play01:50

into south korea now yeah this is just a

play01:53

strategic decision and you're the pm so

play01:55

now we want the uh a good data set to be

play01:58

chosen for the the team

play02:01

okay cool uh do you mind if i just take

play02:03

like 10 seconds just to like kind of

play02:05

construct my phone a little bit here no

play02:07

feel free

play02:13

okay so i would say i mean i think off

play02:15

the bat i think i'm definitely seeing at

play02:17

least kind of like three different

play02:19

factors that we should consider here um

play02:21

the first one being kind of like just

play02:23

you know data quality in the sense of

play02:25

like you know how good is the data

play02:26

meaning like how accurate is the data

play02:29

um are we are we missing anything that

play02:32

like

play02:33

when we talk about businesses and

play02:34

streets for instance like how how

play02:36

closely does that map to reality

play02:38

basically play it

play02:40

um then we have kind of like i was

play02:42

calling the business side of things

play02:43

right like

play02:44

how much are we paying for that um

play02:47

[Music]

play02:48

what what does the kind of like

play02:50

contract look like right this is kind of

play02:52

like like are we committing for like 50

play02:54

years uh this is a yearly renewal thing

play02:57

um these kind of things

play02:59

uh and then finally kind of like a

play03:01

non-functional aspect of kind of like

play03:04

how would the integration look like um

play03:06

what about light licensing

play03:08

can we expect kind of like you know

play03:10

ongoing updates or is this kind of just

play03:12

like a

play03:13

revival data

play03:15

one time and then we have to we have to

play03:17

keep maintaining it so um so i think i

play03:19

think

play03:20

these are these are all kind of factors

play03:21

to consider but um given given that we

play03:24

are google in this scenario um

play03:27

especially

play03:28

google maps i think

play03:30

i'm like i'm gonna make a bit of a

play03:31

little bit of an assumption here but i

play03:33

would imagine that like being google

play03:34

like for instance the cost doesn't

play03:36

really matter right like you know

play03:38

google google typically really cares

play03:39

about like just providing the best

play03:41

experience to their users and i can't

play03:43

imagine that they are gonna be very

play03:45

price sensitive about you know

play03:47

you know which vendor to pick i think

play03:48

they always go with whoever provides the

play03:50

best data um leading to the best user

play03:52

experience here

play03:54

similarly in this kind of like

play03:55

non-functional thing like when we talk

play03:57

about okay how easy is it to integrate

play03:59

um

play04:00

you know given that google is operating

play04:02

at such a big scale like i think even if

play04:04

there was a vendor that like you know

play04:06

maybe provides the data in a slightly

play04:08

different format than what the rest of

play04:09

google maps does today

play04:11

um i don't think that that would also be

play04:13

a huge

play04:15

kind of factor because google google can

play04:17

probably just lift that thing and kind

play04:19

of okay like convert it into whatever

play04:21

format they needed and um and kind of

play04:23

like provide it in a reliable way

play04:27

one thing i didn't mention here for for

play04:28

non-functional also reliability right

play04:30

like um how like like if if we are

play04:32

licensing this maps data

play04:35

and um you're using it directly from the

play04:37

vendor how reliable is it in terms of

play04:39

uptime um etc but i think all these

play04:42

things um google could just basically

play04:45

provide that for the vendor instead

play04:47

right like google google maps is pretty

play04:49

reliable so i think they could just

play04:50

switch all these infrastructure

play04:53

um like all these factors regarding

play04:55

infrastructure they could just switch it

play04:56

over to google infrastructure um so i

play04:58

think really the most important factor

play04:59

here

play05:00

would be data quality so i think in

play05:02

terms of like you know choosing

play05:04

choosing which which vendor we are going

play05:05

with i think data quality should be the

play05:08

single

play05:08

biggest driver in terms of like you know

play05:10

the value in that data because like i

play05:12

said all the other factors i think

play05:14

google can fairly easily or fairly

play05:16

straightforward the

play05:18

um you know solve these these problems

play05:22

in in terms of big pillars that you're

play05:24

looking at is this like some uh

play05:26

unordered list that you just came up

play05:27

with or is this a ranked list

play05:29

um yeah i mean i i it started as an

play05:31

unordered list um i think if if we were

play05:33

to rank it by kind of like importance

play05:35

and priorities like i said i think data

play05:37

quality should be the biggest deciding

play05:39

factor because that's gonna

play05:40

um you know be the main driver for like

play05:43

what the user experience in the end will

play05:45

be

play05:45

um and then i think the next one would

play05:47

be this non-functional aspect but again

play05:49

i think

play05:50

um if that was something that a

play05:53

particular vendor was very bad at i

play05:55

think google could totally with the

play05:56

infrastructure just just uh basically

play05:59

um you know compromise on that so

play06:04

um yeah and then you you mentioned that

play06:07

there's the cost there so for the sake

play06:08

of time let's just dive deeper into the

play06:10

data quality here so can you tell me

play06:12

more about what you would look for in

play06:13

the data quality yeah yeah so um i think

play06:16

what we should start with here is kind

play06:18

of like

play06:19

um just get crisp on like what the data

play06:21

is actually going to be used for and

play06:22

like what the what the user is expecting

play06:24

as well and i think then we can talk

play06:25

about okay what kind of what metrics and

play06:28

what kind of characteristics about the

play06:30

data we would be looking for um if that

play06:31

makes sense to you so here i think the

play06:35

one one key question to figure out

play06:36

whether whether the data quality is good

play06:38

or not is kind of like to think about

play06:39

like what is this data used for right

play06:41

um

play06:43

so we um we kind of talked about

play06:47

we kind of talked about it's kind of

play06:48

like in google maps and um as in

play06:51

as in kind of like

play06:53

assumption here i'm just going to say i

play06:55

think i think google maps kind of like

play06:56

has has two major use cases maybe three

play07:00

like a third one being smaller so the

play07:02

first one i see is definitely kind of

play07:03

like the typical

play07:05

um i i need to go from a to b i need to

play07:07

look up directions

play07:09

um

play07:10

whether that's driving or that's like

play07:12

transit or like walking directions um i

play07:14

just want to know how i get from a to b

play07:16

right so

play07:17

here i would just call that directions

play07:18

and then i think the second one is kind

play07:20

of like

play07:21

um i'm looking for

play07:23

for for for places for like businesses

play07:26

like like i want to look up a restaurant

play07:28

in my area and then like i'm kind of

play07:31

looking for a list of all the places

play07:32

that are better that are around me so

play07:34

here

play07:35

um i would say looking up

play07:37

businesses

play07:40

places or just like just like pois

play07:42

basically

play07:44

um

play07:45

and then when i say looking our business

play07:46

is probably also like information about

play07:48

them right so like i'm i'm interested in

play07:49

their opening hours i'm interested in

play07:51

uh their phone number maybe because i

play07:53

want to call them et cetera and then the

play07:55

third one i was thinking about i don't i

play07:56

would assume it's not a huge one but

play07:58

it's something along the lines of like

play08:00

more like fun things like you know i

play08:02

think um especially google maps people

play08:04

like to like oh this is my house and

play08:05

this is my home address and they wanna

play08:07

they wanna see that on google maps um

play08:09

but i would say that's maybe that's

play08:11

maybe more something for like

play08:13

like street view and satellite imaging

play08:16

which google would

play08:17

uh you know

play08:18

have have already or do on their own in

play08:20

different ways so i think

play08:21

it makes sense uh for us to focus on

play08:23

these two use cases if you if you agree

play08:27

yeah for sure um can you yeah feel free

play08:29

to dive deeper into these if you can

play08:32

sure yeah so um

play08:34

let's let's start maybe with kind of

play08:35

like um

play08:37

the

play08:38

the businesses part so i think here what

play08:41

when we talk about data quality if i'm

play08:43

looking at businesses i'm searching for

play08:44

businesses um i would say there are a

play08:46

few things that are important for us

play08:48

right so i think off your bed i would

play08:49

say

play08:50

um

play08:52

you you definitely have something along

play08:53

the lines of like precision and recall

play08:55

for the data right so like if i um

play08:58

if i am looking for um a business so i

play09:01

imagine i i'm in south korea i'm

play09:03

searching for restaurants in my area

play09:05

um precision in that case would be

play09:08

you know the results that we are giving

play09:10

you the places where we that we give you

play09:12

based on the the vendor's maps data

play09:15

are they are they actually restaurants

play09:16

right like are we giving you the correct

play09:19

answer

play09:20

so um do we find or do we return the

play09:25

right places

play09:27

but then also um

play09:29

recall here is kind of like more like a

play09:31

coverage thing right like how like how

play09:33

often are we able to actually give you

play09:35

restaurants

play09:36

right so i think i think here straight

play09:38

away we kind of like already have two

play09:41

metrics that we probably would want to

play09:42

measure for all these different vendors

play09:44

and compare against

play09:46

that we that we want to maximize so

play09:49

how often do we return

play09:51

places or like results

play09:56

um

play09:58

let me think about other use cases but i

play10:01

think

play10:02

other high level things um

play10:05

i mean i think i think coverage in

play10:06

general is probably something that we

play10:08

should um we should consider here as

play10:10

well so um i mean just to take a step

play10:12

back i think on a high level what we

play10:13

want to do is basically some sort of

play10:15

measurement between

play10:17

how closely is

play10:19

this maps data from this vendor

play10:21

um

play10:22

like how closely is that reflecting the

play10:24

reality right like kind of like the

play10:26

the ground truth um basically

play10:29

um and then that's kind of what we want

play10:31

to compare again so i think what what

play10:32

what i'm trying to think think about now

play10:34

is really

play10:35

what is the best way to kind of like

play10:37

measure that that difference between

play10:39

this is the maps data and this is the

play10:41

reality right

play10:43

um

play10:44

so here i already mentioned like

play10:45

something like procession something like

play10:46

recall

play10:47

um uh kind of like helpful

play10:50

um kind of starting points

play10:52

um

play10:55

i think

play10:57

i think

play10:58

let's just start with these maybe so um

play11:00

if i wanted to figure out um

play11:03

you know how how well is um

play11:05

[Music]

play11:07

how precise is my data in the sense that

play11:09

like

play11:10

um you know

play11:11

all the places that i have in my ins

play11:13

that i'm getting from my vendor are

play11:15

these correct right so obviously are

play11:16

these uh in the right location are we uh

play11:19

our opening hours correct um uh uh you

play11:21

know is the category of the place

play11:23

correct like am i labeling a restaurant

play11:26

as a bar and vice versa things like

play11:27

these

play11:30

how how could i measure this um or like

play11:32

how could i find out

play11:34

what the word difference here is

play11:36

so i think

play11:39

i think what we

play11:41

what we could definitely do

play11:43

is something where we could kind of like

play11:45

just try to basically

play11:47

um

play11:48

almost run and run an ap test right so

play11:50

like what i would imagine is kind of

play11:51

like you could um i mean

play11:53

on the implementation side it's maybe

play11:55

maybe a lot of overhead but like just

play11:57

conceptually what we could do is like we

play12:00

take two different um you know data sets

play12:02

from different vendors

play12:04

and then um you know just just give

play12:06

give a small percentage of users one one

play12:09

set of data and another population

play12:11

doesn't have a user population another

play12:13

set of data but we can compare metrics

play12:15

um so

play12:16

i would say here we can basically do

play12:17

like an

play12:19

a b test

play12:21

using

play12:24

different

play12:25

data sets

play12:28

um

play12:29

before i dive dive further into that i

play12:31

think another higher level approach

play12:33

could also be something like more

play12:37

eval based or like human radar space so

play12:40

what i'm imagining here could be

play12:41

something like

play12:42

um can we just kind of like

play12:45

can can we just get get some golden data

play12:47

where we know for whatever reason that

play12:49

this is this this is reality right and

play12:52

then we can kind of see okay like what

play12:54

what does that data set say about this

play12:57

place right so um here what i'm

play12:59

imagining almost is kind of like maybe

play13:00

we could have

play13:01

operators or like human evolves um

play13:06

just like gather some golden data so you

play13:09

could basically go out there and say hey

play13:10

like this business is here that's its

play13:12

name these are the opening hours and

play13:14

it's kind of manually um you collect

play13:16

that golden data um

play13:18

it's not very scalable but we we we just

play13:20

need a bit kind of like statistically

play13:22

significant

play13:23

a sample size basically and then we

play13:25

could go into the data set and then say

play13:28

what does the data set say uh to tell us

play13:30

about this particular place right does

play13:32

it even exist and then

play13:34

is the data is the data matching what we

play13:36

collected um yeah thanks for flagging

play13:39

that um approach but yeah let's continue

play13:40

with the a

play13:42

a b test here okay yeah so um let's say

play13:44

we run an a b test right so i think here

play13:46

the questions are really kind of like

play13:48

what are the metrics that we would want

play13:49

to um

play13:51

um

play13:51

like kind of monitor right

play13:53

um

play13:55

so i think um

play13:58

i think the basically here um the way i

play14:01

mentioned this av test is kind of like

play14:02

users would would uh you know go to

play14:04

google maps do do their thing and then

play14:06

we we showed them that data set and like

play14:08

i said in the beginning the the typical

play14:10

things that people would do is kind of

play14:11

looking at businesses right so here i

play14:13

think uh like the it's pretty much like

play14:17

a search quality problem right so like i

play14:19

am looking for restaurants in my area

play14:21

i'm getting a set of uh results for that

play14:24

and now we just need to figure out how

play14:25

relevant were these results right and i

play14:27

think one way to do that is certainly

play14:29

looking at interaction metrics so how

play14:32

how often are people you know

play14:33

interacting with the results that that

play14:35

we are giving them

play14:36

um how often are they

play14:38

kind of like

play14:40

abandoning that search and maybe try

play14:41

something new like issuing another

play14:43

search or like maybe doing like a

play14:45

refinement to their to their query um

play14:49

i feel like these are all indicators

play14:50

that like the results that we gave to

play14:52

them were not really

play14:54

relevant or not really helpful right so

play14:56

i think off the bat that's that these

play14:58

are definitely things that i would i

play14:59

would measure here so like interaction

play15:01

rates

play15:02

um as well as like abandonments

play15:06

and um kind of refinements etc

play15:10

so how would you uh understand if a user

play15:13

really got value from the

play15:15

their uh interaction that's that's a

play15:18

really that's a really good question so

play15:19

i think um

play15:21

i mean if you think about it what the

play15:22

user really wants is they want to find

play15:24

relevant places um that that because

play15:27

it's also in the real world that they

play15:29

would want to visit as well right so um

play15:31

i think here like i'm not sure what what

play15:33

google maps kind of like success metric

play15:35

is uh exactly but um i could imagine

play15:38

that we would want something like

play15:40

um did you actually go to that place

play15:42

afterwards for instance right like you

play15:44

could imagine that that might be

play15:46

something like like a real conversion

play15:48

basically right so like it's one thing

play15:49

to like actually open a result and maybe

play15:52

um

play15:53

you know look at it and potentially

play15:55

like interacting with it um but then

play15:58

ultimately i think people use google

play16:00

maps because they want to find places in

play16:01

the real world that they go to right so

play16:04

um it could definitely be a measurement

play16:05

of like how often do you actually go to

play16:08

the place that we showed you here right

play16:11

um and then a proxy to that could be um

play16:13

like in google maps how often do you

play16:14

kind of ask for directions there um how

play16:17

often do you call that place how often

play16:19

do you like

play16:20

like basically interact with it or

play16:22

engage with with that place um so you

play16:24

can you can probably also like save

play16:26

places right as you can you can put a

play16:28

star on them so i think these these kind

play16:30

of engagements um could also be

play16:33

uh be a proxy towards that but i think

play16:35

ultimately

play16:36

like i said maybe we can we can really

play16:38

figure out okay

play16:39

if you search for a place how often do

play16:41

you actually go end up going to that

play16:42

place afterwards yeah that makes sense

play16:44

how often you go there or how often you

play16:45

take a screenshot how often you share

play16:47

share a link yeah something like that so

play16:50

uh let's talk quickly about trade-offs

play16:51

here so let's say that as you're

play16:52

evaluating these different metrics one

play16:55

set gives you really good precision but

play16:57

then another set gives you really good

play16:58

recall so how would you choose the

play17:01

the trade-off here yeah that's that's a

play17:03

really good question i think i think um

play17:05

at the end of the day i think it comes

play17:07

down to kind of like what uh what is

play17:09

google maps um ultimate goal or like

play17:11

success looks like like north star

play17:13

metric basically

play17:14

um

play17:15

i think there will be definitely

play17:17

trade-offs um especially like the

play17:18

precision versus recall trade-off is a

play17:19

very typical one i think what we didn't

play17:21

talk about much is kind of like also

play17:23

things like you know

play17:25

um big cities versus rural right like

play17:28

you might have data sets where like it's

play17:29

it's really great in the big cities but

play17:31

it's terrible and kind of rural areas

play17:34

um

play17:35

so in terms of like how i would make the

play17:38

trade-off here i think really at the end

play17:39

of the day um

play17:41

i would imagine like you know google

play17:42

maps has a nostalgic that we're trying

play17:44

to optimize and um

play17:48

like we would probably have to have to

play17:49

make a call there and saying okay like

play17:51

um this this is closer to the

play17:55

um to the to the ultimate goal of google

play17:57

maps and the success metric like

play17:59

nostalgia metric of google maps um that

play18:02

being said um

play18:03

i wonder if there's something where we

play18:05

can almost make a compromise of like you

play18:07

know can we kind of merge the multiple

play18:09

sets together or can we actually start

play18:11

with one one vendor or like one set data

play18:14

set and then kind of like add things

play18:16

that will uh kind of fill the gaps

play18:18

almost right so i think on google maps

play18:19

is very typical that like for instance

play18:21

users can actually report problems right

play18:24

um so like um like opening up for hours

play18:26

for instance like i i can go to google

play18:28

maps and tell hey like peace opening

play18:30

hours are wrong so um potentially

play18:33

if we if we can identify those gaps we

play18:36

can actually

play18:37

complement like you know the data set

play18:39

that we get from the vendor with like

play18:40

more initiatives in this kind of like

play18:42

user-generated content space um

play18:45

or or you know any any other ways um how

play18:48

we could fill those gaps but i think

play18:50

just just even knowing where the gaps

play18:52

are in the data set that we bought or

play18:53

that we that we decided with i think

play18:55

that's going to be super valuable makes

play18:57

sense yeah figuring out what sort of

play18:58

data is hard for user to generate what

play19:00

sort of data is easy for user generate

play19:02

picking the set that is hard for users

play19:04

to generate and then allowing users to

play19:05

generate the the data

play19:08

yeah opening closing times um yeah great

play19:10

yeah some sort of research study that

play19:12

would be helpful

play19:13

cool uh thanks hal for your your time i

play19:15

think this wraps up our interview for

play19:17

today and i i have some feedback here

play19:20

that i'll i'll share real quick and also

play19:22

love for you to tell me how you might

play19:23

answer the question differently if you

play19:25

got the question again

play19:27

yeah that's that's uh i think i think in

play19:29

hindsight it's always easier right um so

play19:31

i think definitely what i what i noticed

play19:33

as i was going through it um sometimes i

play19:35

was just kind of like jumping ahead a

play19:36

little bit you know like um with

play19:38

precision recall thing it's just like i

play19:40

think it's something that you are

play19:41

trained off and like you want to just

play19:43

say it right but then it wasn't really

play19:44

the right time to say it i feel like i

play19:46

would have been more structured if i

play19:49

uh you know really started with this

play19:50

kind of like okay on a high level what

play19:52

do we want to figure out right like what

play19:54

we really want to figure out is the

play19:55

difference of reality to that to that

play19:57

data set so in in this abstract way

play20:00

um really like even just

play20:02

even just to have have communicated that

play20:04

this is how i feel think about it would

play20:06

have been helpful

play20:08

um

play20:08

as well as like i think in the end we

play20:10

kind of ran more towards okay actually

play20:12

it would be good to know what the what

play20:13

the north star metric or like what the

play20:15

goal of google maps is what that kind of

play20:17

mission is right um so i think that that

play20:20

might have been also helpful to spend

play20:21

some time

play20:22

in the beginning i feel like i feel like

play20:24

i kind of touched on that a little bit

play20:26

in the beginning where i said okay let's

play20:27

think about what this data is going to

play20:28

be used for i mean i think in that part

play20:30

might have been useful to say

play20:33

what is the what is the goal of google

play20:35

maps what is the vision and then also

play20:37

then being able to later tie that back

play20:39

to

play20:40

um what i said earlier right i think i

play20:43

think now in hindsight i feel like it

play20:45

was it was not very

play20:48

um

play20:49

like like there was not really a kind of

play20:50

like way to wrap it up right like you

play20:52

didn't close a circle of like hey like

play20:54

in in the beginning i talked about this

play20:56

and that's why that's why um that's why

play20:58

i'm making this decision here um because

play21:00

remember in the beginning i said this is

play21:02

the goal this is the vision i feel like

play21:04

that was definitely missing

play21:05

yeah for sure i think definitely um

play21:07

refining the mission and the goal in the

play21:09

beginning would help um but overall this

play21:11

was a in my opinion a great mock

play21:13

interview because i was able to at least

play21:14

get the signals that you'd be a good

play21:16

person to work with if you were on my

play21:17

team and just zooming out and thinking

play21:20

about your approach to this i really

play21:21

liked how you were really use case

play21:23

driven when you were diving into the

play21:25

data quality about looking at what you

play21:27

would look for in data quality and you

play21:30

the structure you had was kind of like

play21:32

breadth and then depth so you had this

play21:34

breadth of outlining things that you

play21:35

would consider and then you did a really

play21:37

good job diving deep and drilling down

play21:39

into uh certain parts of the data

play21:41

quality factor that you wanted to

play21:43

investigate

play21:44

so overall really enjoyed this mock

play21:46

interview with you hal thanks for your

play21:47

time and for the viewers i hope you got

play21:49

some value from this video and good luck

play21:51

with your upcoming pm interview

play21:54

thanks so much for watching don't forget

play21:56

to hit the like and subscribe buttons

play21:58

below to let us know that this video is

play22:00

valuable for you and of course check out

play22:02

hundreds more videos just like this at

play22:05

tryexponent.com

play22:07

thanks for watching and good luck on

play22:09

your upcoming interview

play22:14

[Music]

Rate This
β˜…
β˜…
β˜…
β˜…
β˜…

5.0 / 5 (0 votes)

Related Tags
Google MapsSouth KoreaVendor SelectionProduct ManagementData QualityBusiness StrategyInterview MockTech IndustryUser ExperienceMarket ExpansionStrategic Decision