Google PM Interview: Google Maps Korea Launch
Summary
TLDRIn this mock product management interview, Hal, an engineering manager with previous experience at Google, discusses strategies for Google Maps' potential launch in South Korea. He evaluates factors such as data quality, cost, and integration challenges when choosing between local vendors for map datasets. Hal emphasizes the importance of precision and recall in business search results and suggests A/B testing and user-generated content to refine data accuracy. The conversation highlights a user-centric approach to product management, focusing on the ultimate goal of providing the best user experience.
Takeaways
- π Hal, an engineering manager at Cressicore and former PM at Google, discusses choosing a map data vendor for Google Maps' launch in South Korea.
- π The decision-making process should consider data quality, cost, and non-functional aspects like integration and updates.
- π Data quality is crucial for accuracy and user experience, including precision and recall of business listings and directions.
- π° Cost may not be a primary concern for Google, which prioritizes the best user experience over vendor pricing.
- π Non-functional considerations include the ease of integration, licensing, and the reliability of ongoing data updates.
- π Hal suggests using A/B testing with user interaction metrics to evaluate different data sets' effectiveness.
- π Metrics to monitor include interaction rates, search abandonments, and query refinements indicating user satisfaction.
- π’ Google's infrastructure and user engagement features, like reporting incorrect data, can compensate for initial data set deficiencies.
- π The ultimate goal of Google Maps should guide the trade-off decisions between precision and recall of data.
- π€ Hal reflects on the importance of clearly defining the mission and goals at the outset and tying them back to decision-making later.
- π The interview concludes with feedback emphasizing the structured and use-case driven approach Hal took in addressing the problem.
Q & A
What is the main topic of the mock interview in the video transcript?
-The main topic of the mock interview is about Google Maps wanting to launch in South Korea and the decision-making process for choosing a local vendor for map data sets.
Who is Hal and what is his background according to the transcript?
-Hal is an engineering manager at a startup called CressiCore. He was previously a product manager (PM) at Google for five years, working on Google Assistant.
What are the three factors Hal considers when choosing a map data vendor for Google Maps in South Korea?
-Hal considers data quality, the business aspect (including cost and contract terms), and non-functional aspects such as integration ease, licensing, and data reliability.
Why does Hal believe that data quality is the most important factor in choosing a vendor?
-Hal believes data quality is the most important factor because it directly affects the user experience and Google's ability to provide the best service to its users.
What does Hal mean by 'precision and recall' in the context of map data quality?
-In the context of map data quality, 'precision' refers to the accuracy of the data, ensuring that the places listed are indeed what the user is searching for. 'Recall' refers to the coverage of the data, ensuring that the data set includes all relevant places.
How does Hal suggest measuring the difference between the map data and reality?
-Hal suggests using an A/B test with different data sets for a portion of users and comparing metrics like interaction rates, abandonments, and refinements. He also mentions the possibility of using human evaluators to gather 'golden data' for comparison.
What are some of the metrics Hal suggests monitoring during an A/B test of map data?
-Hal suggests monitoring interaction rates, abandonments, and refinements as indicators of how relevant the search results are to the users.
How does Hal define the success metric for Google Maps in the context of the interview?
-Hal suggests that the success metric for Google Maps could be whether users actually visit the places they find through the app, as it indicates the relevance and usefulness of the map data.
What is Hal's approach to dealing with trade-offs between precision and recall of map data?
-Hal's approach is to understand Google Maps' ultimate goal or 'north star' metric and make a decision based on what aligns best with that goal. He also mentions the possibility of merging multiple data sets to balance precision and recall.
What feedback does the interviewer give to Hal at the end of the mock interview?
-The interviewer appreciates Hal's use-case driven approach and his structured method of outlining considerations and diving deep into data quality. The feedback also includes the suggestion to refine the mission and goals at the beginning and to tie back decisions to the initial vision.
Outlines
πΊοΈ Google Maps Expansion Strategy in South Korea
The video script begins with a hypothetical scenario where Google Maps is considering launching in South Korea and must choose between several local vendors for map data. The interviewee, Hal, introduces himself as an ex-Google PM and current engineering manager. The discussion focuses on the criteria for selecting a map data vendor, emphasizing the importance of data quality, cost, and integration feasibility. Hal suggests that for Google, data quality is paramount to ensure an excellent user experience, while cost and integration challenges are secondary concerns due to Google's resources and capabilities.
π Prioritizing Data Quality for Google Maps Launch
In this paragraph, Hal delves into the factors that determine data quality, particularly precision and recall of business listings. He explains that precision refers to the accuracy of the search results, ensuring that the places listed are indeed what the user is searching for. Recall, on the other hand, is about the comprehensiveness of the data, ensuring that all relevant businesses are included. Hal suggests that Google could potentially use A/B testing with different data sets to measure these metrics and make an informed decision on the vendor that provides the best data quality.
π Measuring Data Quality Through User Interaction
Hal discusses how to measure the effectiveness of map data through user interaction metrics. He suggests monitoring how users engage with search results, including interaction rates, abandonments, and refinements. These metrics can indicate the relevance and usefulness of the data provided. Furthermore, Hal proposes tracking whether users actually visit the places they find on Google Maps, suggesting that this could be a strong indicator of the data's real-world value. He also considers the possibility of using user-generated content to fill gaps in the initial data set.
π€ Balancing Precision and Recall in Data Selection
The script addresses the trade-off between precision and recall in selecting map data. Hal acknowledges that one data set might excel in precision while another might have better recall. He suggests that the decision on which to prioritize depends on Google Maps' ultimate goal or 'north star' metric. Hal also contemplates combining multiple data sets or starting with one and using user contributions to improve it, highlighting the importance of identifying and addressing gaps in the data.
π‘ Reflecting on the Interview Process and Approach
In the final paragraph, Hal reflects on the mock interview process. He admits to jumping ahead at times and not clearly defining Google Maps' mission and goal at the outset. Hal believes that a more structured approach, starting with the mission and tying it back to decision-making later, would have been beneficial. The interviewer appreciates Hal's use-case driven strategy and his methodical dive into data quality, finding the interview valuable and indicating that Hal would be a good team member.
Mindmap
Keywords
π‘Product Management
π‘Google Maps
π‘Vendor
π‘Data Quality
π‘Precision and Recall
π‘A/B Testing
π‘Integration
π‘User Experience
π‘North Star Metric
π‘Trade-offs
π‘User-Generated Content
Highlights
Google Maps is considering launching in South Korea and evaluating local vendors for map data sets.
Interviewee Hal, an engineering manager with previous experience as a PM at Google, discusses the strategic decision-making process for choosing map data vendors.
Three key factors identified for decision-making: data quality, business terms (cost and contract), and integration feasibility.
Data quality is emphasized as the most critical factor due to its direct impact on user experience.
The importance of precision and recall in evaluating map data for business listings is discussed.
Coverage of map data, including its reflection of reality, is considered crucial for data quality.
Google's infrastructure and capabilities are highlighted as potential solutions for integration challenges.
The role of user-generated content in enhancing map data through reporting inaccuracies is mentioned.
A/B testing different data sets with user populations to measure effectiveness is proposed.
Interaction metrics, such as search refinements and abandonments, are suggested as indicators of data relevance.
The concept of 'golden data' collected manually to verify and ensure data accuracy is introduced.
The potential use of user engagement with map results as a proxy for measuring data value is discussed.
Trade-offs between precision and recall in map data are considered, with a focus on Google Maps' ultimate goal.
The idea of merging multiple data sets or complementing initial data with user-generated content is explored.
The interview concludes with feedback on Hal's structured approach and use-case driven strategy in evaluating data quality.
The importance of aligning decision-making with Google Maps' mission and vision is emphasized in the debrief.
The mock interview serves as a valuable example for viewers preparing for product management interviews.
Transcripts
let's imagine that google maps wants to
launch in south korea and they're you're
choosing between several local vendors
and each one has a different map data
set tell me which one you would choose
[Music]
hey everyone welcome back to another
exponent product management mock
interview my name is kevin and on
today's show we have how we're going to
be doing a product management analytical
or strategy type question but before we
jump into that how can you just tell the
audience a little bit about who you are
hey everyone uh so i'm hal i'm currently
an engineering manager in a startup
called cressicore but before that i was
a pm and 8pm at google for five years
working on the google assistant
sweet thanks for your time today so this
is what i like to ask you let's imagine
that google maps wants to launch in
south korea and they're you're choosing
between several local vendors and each
one has a different map data set tell me
which one you would choose
okay so google maps wants to expand
basically in a new country and i'm
choosing between
[Music]
um different vendors for for maps data
yes
okay
um
and then how do i
how do i decide basically
okay so uh maybe just just to just to
clarify but so when we say maps data i
assume that's kind of like you know the
data that i would see in you know the
google maps app and uh so like when i
when i open google maps in south korea
like whatever i see there with the
streets and the
the businesses that that's what i'm
that's what we're referring to here
right
yeah we can start with this assumption
and then if we have time we can expand
upon this definition cool okay yeah
and
um is this specifically like um like
like you mentioned south korea is there
a specific reason why why
google maps is not in south korea yet or
like
why why are we why why are we expanding
into south korea now yeah this is just a
strategic decision and you're the pm so
now we want the uh a good data set to be
chosen for the the team
okay cool uh do you mind if i just take
like 10 seconds just to like kind of
construct my phone a little bit here no
feel free
okay so i would say i mean i think off
the bat i think i'm definitely seeing at
least kind of like three different
factors that we should consider here um
the first one being kind of like just
you know data quality in the sense of
like you know how good is the data
meaning like how accurate is the data
um are we are we missing anything that
like
when we talk about businesses and
streets for instance like how how
closely does that map to reality
basically play it
um then we have kind of like i was
calling the business side of things
right like
how much are we paying for that um
[Music]
what what does the kind of like
contract look like right this is kind of
like like are we committing for like 50
years uh this is a yearly renewal thing
um these kind of things
uh and then finally kind of like a
non-functional aspect of kind of like
how would the integration look like um
what about light licensing
can we expect kind of like you know
ongoing updates or is this kind of just
like a
revival data
one time and then we have to we have to
keep maintaining it so um so i think i
think
these are these are all kind of factors
to consider but um given given that we
are google in this scenario um
especially
google maps i think
i'm like i'm gonna make a bit of a
little bit of an assumption here but i
would imagine that like being google
like for instance the cost doesn't
really matter right like you know
google google typically really cares
about like just providing the best
experience to their users and i can't
imagine that they are gonna be very
price sensitive about you know
you know which vendor to pick i think
they always go with whoever provides the
best data um leading to the best user
experience here
similarly in this kind of like
non-functional thing like when we talk
about okay how easy is it to integrate
um
you know given that google is operating
at such a big scale like i think even if
there was a vendor that like you know
maybe provides the data in a slightly
different format than what the rest of
google maps does today
um i don't think that that would also be
a huge
kind of factor because google google can
probably just lift that thing and kind
of okay like convert it into whatever
format they needed and um and kind of
like provide it in a reliable way
one thing i didn't mention here for for
non-functional also reliability right
like um how like like if if we are
licensing this maps data
and um you're using it directly from the
vendor how reliable is it in terms of
uptime um etc but i think all these
things um google could just basically
provide that for the vendor instead
right like google google maps is pretty
reliable so i think they could just
switch all these infrastructure
um like all these factors regarding
infrastructure they could just switch it
over to google infrastructure um so i
think really the most important factor
here
would be data quality so i think in
terms of like you know choosing
choosing which which vendor we are going
with i think data quality should be the
single
biggest driver in terms of like you know
the value in that data because like i
said all the other factors i think
google can fairly easily or fairly
straightforward the
um you know solve these these problems
in in terms of big pillars that you're
looking at is this like some uh
unordered list that you just came up
with or is this a ranked list
um yeah i mean i i it started as an
unordered list um i think if if we were
to rank it by kind of like importance
and priorities like i said i think data
quality should be the biggest deciding
factor because that's gonna
um you know be the main driver for like
what the user experience in the end will
be
um and then i think the next one would
be this non-functional aspect but again
i think
um if that was something that a
particular vendor was very bad at i
think google could totally with the
infrastructure just just uh basically
um you know compromise on that so
um yeah and then you you mentioned that
there's the cost there so for the sake
of time let's just dive deeper into the
data quality here so can you tell me
more about what you would look for in
the data quality yeah yeah so um i think
what we should start with here is kind
of like
um just get crisp on like what the data
is actually going to be used for and
like what the what the user is expecting
as well and i think then we can talk
about okay what kind of what metrics and
what kind of characteristics about the
data we would be looking for um if that
makes sense to you so here i think the
one one key question to figure out
whether whether the data quality is good
or not is kind of like to think about
like what is this data used for right
um
so we um we kind of talked about
we kind of talked about it's kind of
like in google maps and um as in
as in kind of like
assumption here i'm just going to say i
think i think google maps kind of like
has has two major use cases maybe three
like a third one being smaller so the
first one i see is definitely kind of
like the typical
um i i need to go from a to b i need to
look up directions
um
whether that's driving or that's like
transit or like walking directions um i
just want to know how i get from a to b
right so
here i would just call that directions
and then i think the second one is kind
of like
um i'm looking for
for for for places for like businesses
like like i want to look up a restaurant
in my area and then like i'm kind of
looking for a list of all the places
that are better that are around me so
here
um i would say looking up
businesses
places or just like just like pois
basically
um
and then when i say looking our business
is probably also like information about
them right so like i'm i'm interested in
their opening hours i'm interested in
uh their phone number maybe because i
want to call them et cetera and then the
third one i was thinking about i don't i
would assume it's not a huge one but
it's something along the lines of like
more like fun things like you know i
think um especially google maps people
like to like oh this is my house and
this is my home address and they wanna
they wanna see that on google maps um
but i would say that's maybe that's
maybe more something for like
like street view and satellite imaging
which google would
uh you know
have have already or do on their own in
different ways so i think
it makes sense uh for us to focus on
these two use cases if you if you agree
yeah for sure um can you yeah feel free
to dive deeper into these if you can
sure yeah so um
let's let's start maybe with kind of
like um
the
the businesses part so i think here what
when we talk about data quality if i'm
looking at businesses i'm searching for
businesses um i would say there are a
few things that are important for us
right so i think off your bed i would
say
um
you you definitely have something along
the lines of like precision and recall
for the data right so like if i um
if i am looking for um a business so i
imagine i i'm in south korea i'm
searching for restaurants in my area
um precision in that case would be
you know the results that we are giving
you the places where we that we give you
based on the the vendor's maps data
are they are they actually restaurants
right like are we giving you the correct
answer
so um do we find or do we return the
right places
but then also um
recall here is kind of like more like a
coverage thing right like how like how
often are we able to actually give you
restaurants
right so i think i think here straight
away we kind of like already have two
metrics that we probably would want to
measure for all these different vendors
and compare against
that we that we want to maximize so
how often do we return
places or like results
um
let me think about other use cases but i
think
other high level things um
i mean i think i think coverage in
general is probably something that we
should um we should consider here as
well so um i mean just to take a step
back i think on a high level what we
want to do is basically some sort of
measurement between
how closely is
this maps data from this vendor
um
like how closely is that reflecting the
reality right like kind of like the
the ground truth um basically
um and then that's kind of what we want
to compare again so i think what what
what i'm trying to think think about now
is really
what is the best way to kind of like
measure that that difference between
this is the maps data and this is the
reality right
um
so here i already mentioned like
something like procession something like
recall
um uh kind of like helpful
um kind of starting points
um
i think
i think
let's just start with these maybe so um
if i wanted to figure out um
you know how how well is um
[Music]
how precise is my data in the sense that
like
um you know
all the places that i have in my ins
that i'm getting from my vendor are
these correct right so obviously are
these uh in the right location are we uh
our opening hours correct um uh uh you
know is the category of the place
correct like am i labeling a restaurant
as a bar and vice versa things like
these
how how could i measure this um or like
how could i find out
what the word difference here is
so i think
i think what we
what we could definitely do
is something where we could kind of like
just try to basically
um
almost run and run an ap test right so
like what i would imagine is kind of
like you could um i mean
on the implementation side it's maybe
maybe a lot of overhead but like just
conceptually what we could do is like we
take two different um you know data sets
from different vendors
and then um you know just just give
give a small percentage of users one one
set of data and another population
doesn't have a user population another
set of data but we can compare metrics
um so
i would say here we can basically do
like an
a b test
using
different
data sets
um
before i dive dive further into that i
think another higher level approach
could also be something like more
eval based or like human radar space so
what i'm imagining here could be
something like
um can we just kind of like
can can we just get get some golden data
where we know for whatever reason that
this is this this is reality right and
then we can kind of see okay like what
what does that data set say about this
place right so um here what i'm
imagining almost is kind of like maybe
we could have
operators or like human evolves um
just like gather some golden data so you
could basically go out there and say hey
like this business is here that's its
name these are the opening hours and
it's kind of manually um you collect
that golden data um
it's not very scalable but we we we just
need a bit kind of like statistically
significant
a sample size basically and then we
could go into the data set and then say
what does the data set say uh to tell us
about this particular place right does
it even exist and then
is the data is the data matching what we
collected um yeah thanks for flagging
that um approach but yeah let's continue
with the a
a b test here okay yeah so um let's say
we run an a b test right so i think here
the questions are really kind of like
what are the metrics that we would want
to um
um
like kind of monitor right
um
so i think um
i think the basically here um the way i
mentioned this av test is kind of like
users would would uh you know go to
google maps do do their thing and then
we we showed them that data set and like
i said in the beginning the the typical
things that people would do is kind of
looking at businesses right so here i
think uh like the it's pretty much like
a search quality problem right so like i
am looking for restaurants in my area
i'm getting a set of uh results for that
and now we just need to figure out how
relevant were these results right and i
think one way to do that is certainly
looking at interaction metrics so how
how often are people you know
interacting with the results that that
we are giving them
um how often are they
kind of like
abandoning that search and maybe try
something new like issuing another
search or like maybe doing like a
refinement to their to their query um
i feel like these are all indicators
that like the results that we gave to
them were not really
relevant or not really helpful right so
i think off the bat that's that these
are definitely things that i would i
would measure here so like interaction
rates
um as well as like abandonments
and um kind of refinements etc
so how would you uh understand if a user
really got value from the
their uh interaction that's that's a
really that's a really good question so
i think um
i mean if you think about it what the
user really wants is they want to find
relevant places um that that because
it's also in the real world that they
would want to visit as well right so um
i think here like i'm not sure what what
google maps kind of like success metric
is uh exactly but um i could imagine
that we would want something like
um did you actually go to that place
afterwards for instance right like you
could imagine that that might be
something like like a real conversion
basically right so like it's one thing
to like actually open a result and maybe
um
you know look at it and potentially
like interacting with it um but then
ultimately i think people use google
maps because they want to find places in
the real world that they go to right so
um it could definitely be a measurement
of like how often do you actually go to
the place that we showed you here right
um and then a proxy to that could be um
like in google maps how often do you
kind of ask for directions there um how
often do you call that place how often
do you like
like basically interact with it or
engage with with that place um so you
can you can probably also like save
places right as you can you can put a
star on them so i think these these kind
of engagements um could also be
uh be a proxy towards that but i think
ultimately
like i said maybe we can we can really
figure out okay
if you search for a place how often do
you actually go end up going to that
place afterwards yeah that makes sense
how often you go there or how often you
take a screenshot how often you share
share a link yeah something like that so
uh let's talk quickly about trade-offs
here so let's say that as you're
evaluating these different metrics one
set gives you really good precision but
then another set gives you really good
recall so how would you choose the
the trade-off here yeah that's that's a
really good question i think i think um
at the end of the day i think it comes
down to kind of like what uh what is
google maps um ultimate goal or like
success looks like like north star
metric basically
um
i think there will be definitely
trade-offs um especially like the
precision versus recall trade-off is a
very typical one i think what we didn't
talk about much is kind of like also
things like you know
um big cities versus rural right like
you might have data sets where like it's
it's really great in the big cities but
it's terrible and kind of rural areas
um
so in terms of like how i would make the
trade-off here i think really at the end
of the day um
i would imagine like you know google
maps has a nostalgic that we're trying
to optimize and um
like we would probably have to have to
make a call there and saying okay like
um this this is closer to the
um to the to the ultimate goal of google
maps and the success metric like
nostalgia metric of google maps um that
being said um
i wonder if there's something where we
can almost make a compromise of like you
know can we kind of merge the multiple
sets together or can we actually start
with one one vendor or like one set data
set and then kind of like add things
that will uh kind of fill the gaps
almost right so i think on google maps
is very typical that like for instance
users can actually report problems right
um so like um like opening up for hours
for instance like i i can go to google
maps and tell hey like peace opening
hours are wrong so um potentially
if we if we can identify those gaps we
can actually
complement like you know the data set
that we get from the vendor with like
more initiatives in this kind of like
user-generated content space um
or or you know any any other ways um how
we could fill those gaps but i think
just just even knowing where the gaps
are in the data set that we bought or
that we that we decided with i think
that's going to be super valuable makes
sense yeah figuring out what sort of
data is hard for user to generate what
sort of data is easy for user generate
picking the set that is hard for users
to generate and then allowing users to
generate the the data
yeah opening closing times um yeah great
yeah some sort of research study that
would be helpful
cool uh thanks hal for your your time i
think this wraps up our interview for
today and i i have some feedback here
that i'll i'll share real quick and also
love for you to tell me how you might
answer the question differently if you
got the question again
yeah that's that's uh i think i think in
hindsight it's always easier right um so
i think definitely what i what i noticed
as i was going through it um sometimes i
was just kind of like jumping ahead a
little bit you know like um with
precision recall thing it's just like i
think it's something that you are
trained off and like you want to just
say it right but then it wasn't really
the right time to say it i feel like i
would have been more structured if i
uh you know really started with this
kind of like okay on a high level what
do we want to figure out right like what
we really want to figure out is the
difference of reality to that to that
data set so in in this abstract way
um really like even just
even just to have have communicated that
this is how i feel think about it would
have been helpful
um
as well as like i think in the end we
kind of ran more towards okay actually
it would be good to know what the what
the north star metric or like what the
goal of google maps is what that kind of
mission is right um so i think that that
might have been also helpful to spend
some time
in the beginning i feel like i feel like
i kind of touched on that a little bit
in the beginning where i said okay let's
think about what this data is going to
be used for i mean i think in that part
might have been useful to say
what is the what is the goal of google
maps what is the vision and then also
then being able to later tie that back
to
um what i said earlier right i think i
think now in hindsight i feel like it
was it was not very
um
like like there was not really a kind of
like way to wrap it up right like you
didn't close a circle of like hey like
in in the beginning i talked about this
and that's why that's why um that's why
i'm making this decision here um because
remember in the beginning i said this is
the goal this is the vision i feel like
that was definitely missing
yeah for sure i think definitely um
refining the mission and the goal in the
beginning would help um but overall this
was a in my opinion a great mock
interview because i was able to at least
get the signals that you'd be a good
person to work with if you were on my
team and just zooming out and thinking
about your approach to this i really
liked how you were really use case
driven when you were diving into the
data quality about looking at what you
would look for in data quality and you
the structure you had was kind of like
breadth and then depth so you had this
breadth of outlining things that you
would consider and then you did a really
good job diving deep and drilling down
into uh certain parts of the data
quality factor that you wanted to
investigate
so overall really enjoyed this mock
interview with you hal thanks for your
time and for the viewers i hope you got
some value from this video and good luck
with your upcoming pm interview
thanks so much for watching don't forget
to hit the like and subscribe buttons
below to let us know that this video is
valuable for you and of course check out
hundreds more videos just like this at
tryexponent.com
thanks for watching and good luck on
your upcoming interview
[Music]
Browse More Related Video
![](https://i.ytimg.com/vi/l8s9_ScogCk/hq720.jpg)
Google Product Manager Execution Interview: YouTube Watch Time Root Cause Analysis
![](https://i.ytimg.com/vi/FYMdFroNc9E/hq720.jpg)
Health Tech Product Manager Interview: Doctor Search
![](https://i.ytimg.com/vi/DSV-vuvmIro/hq720.jpg)
Flipkart Product Manager Mock Interview: Root Cause Analysis (Razorpay PM)
![](https://i.ytimg.com/vi/F3SIVlcWgUg/hq720.jpg)
Facebook Product Manager Interview: Meaningful Social Metrics
![](https://i.ytimg.com/vi/3Qx9cVRJ06I/hq720.jpg)
Facebook Product Manager Execution Interview: YouTube Goals & Decline
![](https://i.ytimg.com/vi/nbB1H6XpPuU/hq720.jpg)
Netflix Product Manager Interview: Inactive Users
5.0 / 5 (0 votes)