Bell's data modernization journey: executing for the future with Google
Summary
TLDREric from Bell's Enterprise data platform team shares his journey from intern to architect and discusses the company's transition to Google Cloud. He highlights three key mistakes made during their on-premise data management: proliferating data silos, neglecting developer productivity, and poor data governance practices. Eric outlines how Bell is addressing these issues by standardizing on a single data platform, improving developer tools, and adopting data cataloging with Google Cloud's Dataplex.
Takeaways
- đ Bell is Canada's largest telecom provider, serving multiple provinces and regions.
- đčïž The speaker, Eric, works in Bell's Enterprise Data Platform team and has been with the company for 17 years, starting as an intern.
- đ The speaker used his experience in competitive gaming to draw parallels between analyzing mistakes in games and reviewing past data management strategies.
- âïž Bell has recently migrated to Google Cloud and is reviewing past mistakes to avoid repeating them in the new environment.
- đïž The first major mistake identified was the creation of numerous data silos, leading to inefficiencies and challenges in data management.
- đ§ The second mistake was the lack of focus on improving developer productivity, leading to redundant and inefficient coding practices.
- đ ïž Bell is now working on standardizing their data platform, consolidating their data into Google BigQuery, and improving data accessibility.
- đ» The company is also implementing tools like Terraform modules and Backstage to standardize and streamline developer workflows.
- đ Bell plans to create config-driven libraries to reduce redundant coding and improve the efficiency of data pipeline creation.
- đ The third mistake involved poor data governance practices, with metadata managed in Excel and Confluence. Bell is now standardizing on DataPlex for better data cataloging and governance.
Q & A
Who is the speaker, and what is his role at Bell?
-The speaker is Eric, who works in the Enterprise Data Platform team at Bell. He has been with the company for 17 years, starting as an intern and then moving into roles such as Data Engineer and currently, an Architect.
What is Bell, and what regions does it serve?
-Bell is the largest telecommunications provider in Canada, serving regions like Quebec, Ontario, Atlantic provinces, and Manitoba.
Why does the speaker mention playing competitive video games, and how does it relate to the talk?
-The speaker mentions playing competitive video games like Rocket League to draw a parallel between analyzing mistakes in gaming and analyzing mistakes made during Bell's data platform migration. The idea is that reviewing mistakes is crucial for improvement in both contexts.
What were the three major mistakes Bell made with their on-premise data systems?
-The three major mistakes were: 1) Deploying dozens of data environments and creating data silos. 2) Failing to improve developers' productivity and experience. 3) Poor data governance practices, including the management of metadata in Excel spreadsheets and Confluence.
What was the impact of deploying multiple data environments at Bell?
-Deploying multiple data environments led to a complex web of data pipelines, making it difficult for engineers to create value and requiring them to spend a lot of time moving data around instead of solving core problems.
How did Bell plan to address the issue of multiple data environments in their migration to Google Cloud?
-Bell planned to standardize on a single data platform in Google Cloud, specifically using Google BigQuery. This would centralize data and reduce the need for engineers to move data across different environments.
What problems did Bell encounter with developer productivity, and how are they addressing these in the cloud migration?
-Bell found that developers were repeatedly solving the same problems and writing redundant code, particularly in areas like data compaction. To address this, they are building a solid foundation with Terraform modules and using tools like Backstage to standardize and automate processes, improving productivity.
What is Backstage, and how is Bell utilizing it in their cloud migration?
-Backstage is an open-source developer portal created by Spotify that allows engineers to use templates to quickly spin up applications. Bell is using Backstage to simplify and standardize their development processes during their cloud migration.
What issue did Bell face with data governance, and what tool are they using to improve it?
-Bell struggled with managing table metadata using Excel spreadsheets and Confluence, leading to inefficiencies. To improve data governance, they are now standardizing with Google DataPlex to catalog assets and provide a centralized location for data governance.
What is the speaker's main message to the audience regarding their own experiences with large-scale migrations?
-The speaker encourages the audience to learn from Bell's mistakes and shares their experiences to help others avoid similar pitfalls. He invites others with experience in large-scale migrations to reach out and share their learnings, particularly as Bell prepares for a significant migration next month.
Outlines
đ€ Introduction and Background
Eric introduces himself as a member of the Enterprise Data Platform team at Bell, where he has worked for 17 years, starting as an intern. He gives a brief overview of Bell as Canadaâs largest telecom provider, emphasizing its extensive service coverage. Eric then shares a personal anecdote about playing competitive video games like Rocket League, using it as a metaphor to discuss learning from mistakes, which ties into the main theme of his talk: reflecting on past errors in Bellâs data management and sharing three major mistakes made during their transition to Google Cloud.
đïž Mistake 1: Data Silos and Fragmented Systems
Eric delves into the first major mistake made at Bell: the proliferation of data environments and silos over the years. Initially, the goal was to create a unified data lake when implementing Hadoop in 2016. However, the effort was abandoned midway, leading to a fragmented system where engineers worked in isolated tech stacks, resulting in a complex web of data pipelines. This fragmentation hindered efficient data usage and slowed down processes like data monetization. The solution, Eric explains, is to standardize on a single data platform in Google Cloud, consolidating data to streamline access and reduce the need for redundant pipelines.
đ Mistake 2: Neglecting Developer Productivity
The second mistake Eric discusses is the failure to prioritize developer productivity on Bellâs on-premise systems. Developers were burdened with managing infrastructure and repeatedly solving the same problems due to a lack of standardized tools. For example, multiple teams independently developed similar libraries for tasks like data compaction in Hadoop. To address this, Bell is now focused on building a solid foundation in the cloud, starting with standardized Terraform modules and leveraging tools like Backstage, an open-source developer portal created by Spotify, to improve the efficiency of application development and reduce code redundancy.
đ ïž Mistake 3: Poor Data Governance Practices
The third and final mistake highlighted is Bellâs inadequate data governance practices. Eric admits that table metadata was being managed in Excel spreadsheets and Confluence, leading to inefficiencies and confusion among engineers. Moving to the cloud, Bell is now standardizing with Dataplex to properly catalog and manage data assets, ensuring that information is easily accessible and well-organized. This shift aims to eliminate the chaos of managing data across multiple platforms and improve overall data governance. Eric concludes by inviting the audience to share their experiences with large-scale migrations and offering to connect for further discussions.
Mindmap
Keywords
đĄData Silos
đĄCloud Migration
đĄData Governance
đĄDeveloper Productivity
đĄTerraform
đĄBackstage
đĄData Pipelines
đĄETL (Extract, Transform, Load)
đĄData Monetization
đĄData Lake
Highlights
Introduction by Eric, who has worked at Bell for 17 years, starting as an intern and now an architect in the Enterprise Data Platform team.
Bell is the largest telecom provider in Canada, serving regions including Quebec, Ontario, Atlantic provinces, and Manitoba.
Eric's playful icebreaker about competitive video games, sharing his personal interest in Rocket League.
Bell's recent migration to Google Cloud, aiming to learn from past mistakes made on-premises.
Highlighting three major mistakes made by Bell in their data practices: deploying data silos, neglecting developer productivity, and poor data governance.
Mistake 1: Creating multiple data silos over time, leading to fragmented data environments and inefficient data management.
The shift to a standardized data platform on Google BigQuery to centralize data and streamline access.
Mistake 2: Neglecting developer productivity by focusing too much on infrastructure management rather than improving developer experience.
Efforts to modernize by building Terraform modules for standardized naming conventions, best practices, and labeling in the cloud.
Bell's proof of concept with Backstage, an open-source developer portal created by Spotify, to streamline application development.
Mistake 3: Poor data governance practices, including managing metadata in Excel spreadsheets and Confluence.
Adoption of Google DataPlex for standardized data governance, improving metadata management and reducing reliance on outdated methods.
Encouragement to the audience to avoid these common mistakes, emphasizing the importance of learning from Bell's experiences.
Eric's invitation to the audience to share their experiences with large-scale migrations, expressing a desire to learn from others.
Closing remarks with an open invitation for questions and further discussion on the topic.
Transcripts
[Music]
all right hello everyone hopefully
you're all having a great time uh what a
wonderful event right really lots of
nice talk so um I'm Eric um I work in
the Enterprise data platform team in
Bell uh I've been there for the past 17
years uh started there as a intern and
then uh data engineer with the Microsoft
SQL Server stack uh before data was
considered pretty nice right and then
grew as a data engineer with the dupe
and then here I am as an architect right
um so
uh quickly uh what what is Bell right
it's the largest tcom provider in in
Canada uh bigger than the other
competitor right before and uh we
essentially uh we've been uh providing
services to like Quebec Ontario Atlantic
provinces um
Manitoba so uh yeah so essentially today
uh I'm going to just ask a quick
question just to break the eyes right I
see a few younger folks here uh so I
want to ask is anybody here uh plays
online video games like competitive
video games specifically like um uh
League of Legend fortnite Counter Strike
anybody
nobody okay wow we don't have lots of
competitive players here okay okay so I
do I I'll I'll admit it I do and uh I'm
a huge fan of Rocket League personally
and uh the reason um one of the thing
that I've started when I played this
game five years ago is um I would often
like uh come in play my games and then a
bad goal happen I lose the game and then
what do I do just blame the teammate
right who else could it
be so um but it turns out that when you
play quite a bit of of competitive game
one of the things you learn to do is um
watch your only replays right it's the
only way where you're going to be able
to figure out your mistake and then not
reproduce them obviously so um that's
what I did a bit right and the the
reason I'm I'm talking a bit about this
is we've had the opportunity for the
past one year to uh migrate to Google
cloud and then sort of look at what like
what we did wrong on premise like what
are the big mistakes we did on premise
and uh what we should change so that it
doesn't happen again when we're going to
the cloud right so I decided for today's
talk to just put a list of a short list
of three mistakes we did and then I'm
here to share this with you and then I'm
going to share our experience and then
hopefully you'll learn something from
this so the first one and I think it's a
pretty common mistake in the Telecom
industry is we've um essentially
deployed dozens of data environments
data silos through the years and uh this
just grew over time more and more right
so that's number one number two is um we
uh we didn't really take time to improve
our developers productivity and
experience right it was a second thought
for us and then last one but not not the
least it's the our data governance
practices right and tooling that were uh
pretty bad let's say so I'm going to go
through like all these three uh three uh
mistakes we did explain where we went
wrong and what we're going to change by
going to the cloud so first one right I
mentioned are dozens of data warehouse
data environments so um for this one uh
we we started like um obviously we have
lots of Legacy systems in Bell right and
um when we implemented a dupe back in
2016 the premise was well let's let's
build the data Lake let's migrate our
ETL through there and then we'll just
have a single place to consume our data
right uh not exactly what happened okay
so what we did is we actually
implemented a dupe we migrated a lot of
ETL we migrated a lot of data set and
then we just stopped Midway we just
stopped and and then move on to
something else right just focus on value
creation so um so yeah obviously what
happened is all these data Engineers
data Specialists uh were just focusing
the they were in their own Tex stack
right so imagine you have an an engineer
who works in the Microsoft SQL
environment and he plays nice there and
whenever he needs data he's just going
to develop an ETL pipeline to move it to
his own data world so what happened is a
huge spaghetti a mess of data pipeline
all over the place and um it's it's
really long for us right like whenever
you want to create value you want to do
a IML you want to do uh we're we're
doing lots of data monetization in Bell
and obviously this is hard for us
because our Engineers are just focusing
time on not solving this problem they're
focusing on moving the data
around so what we've decided to change
on the cloud is we decided to end it
there and then just standardize on a
single data platform so instead of
having okay well we have orle we're
going to move orle to Google Cloud we
have SQL Server we're going to move that
to Google Cloud let's take the time to
modernize these Legacy environments move
them to Google big query and then once
Engineers have their data all in one
place it's very simple right you if if
if if somebody needs access to the data
a simple I am permission and you're done
nothing else right no need to develop
pipelines to move data around so um so
yeah we're we're really eager to have
more and more data there so that we can
stop just injuring pipelines over and
over
again okay so second one right second
mistake we did is um and I think that's
a really important one right you're
developers
productivity uh so your your developers
your engineers are so important you need
to improve their experience so what we
did on premise is we have a dup we just
manage infrastructure we manage server
we manage Hardware uh and then that's
that's all we do right and we did a
little bit of shell scripting here and
there to create database to um to create
like uh help us with permission but
nothing nothing else really so for us it
was really not a good experience for
developers and what we've realize is
when we uh did an assessment recently
for migrating our adup we've realized we
look at different git repositories look
at the source code and realize wow our
Engineers are just solving the same
problem over and over again right uh
just as an
example um we had uh a
um if you've ever worked with the dup
you might have heard of data compaction
to to compact small files right into
larger files um we essentially had six
separate libraries just to do the same
thing right all these teams they build
the same Library without speaking with
each others and here we are right a huge
mess of lots of of code so we're
essentially going to use that
opportunity to modernize so how are we
going to solve it few things right first
thing we did when we uh migrated is
let's right away build a solid
foundation and that starts with uh for
example building terraform module to
standardize on um anything that's like
um naming conventions standards best
practices labeling so all of that right
we would bake it right in so that people
don't repeat that code over and and over
again second thing is I highly encourage
you look at it we recently did a proof
of concept with the tool called uh
backstage so I'm not sure if anybody
knows what backstage is if you don't
know there's an interesting talk
tomorrow from uh um I think it's HC
Healthcare so um they have an
interesting talk about how they um they
use this tool to help developers It's
actually an open-source um developer
port that was created by Spotify and
then they open source it and then you
can sort of have it a bunch of templates
and then you can uh templae your
applications and then Engineers they go
in a UI they simply fill a form and then
there we are they can spin up
applications very fast right so um after
this proof of concept we've been very
excited with that and I think we're
about to roll it out to more and more
type of templates right
um so yeah so essentially um the other
thing was um that we're going to do to
help our developers is we've realized
when we did our adup migration that
there's um I think there was hundreds of
data pipeline almost a thousand
pipelines that were just strictly for um
ingesting relational database data MySQL
Orco postgres db2 right and all of
Engineers um they would just copypaste
code over and over again so what are we
doing for that well um we're going to
make config driven uh libraries to do
that right to help Engineers be more
productive and avoid having to write
custom code all the time so what with
all of these right it should make our
engineer way more productive on the
cloud and and stop just writing code
focus on other things
okay so last last thing but not least is
um the last mistake I want to mention is
our data governance practice okay so I'm
sure we're lots of us are in the same
boat so I'll admit it we do manage uh
table metadata in Excel spreadsheets and
Confluence okay I'll say it and I'm sure
lots of you maybe uh have the same
situation so the first thing we did by
going to Cloud is go standardize with
data Plex right make sure our Engineers
are um are just um cataloging their
assets their tables their pii
hspi everything is is in in there right
so we can stop having people asking
questions on slack email where is this
table where is it does it exist well you
you will have it somewhere right it will
be catalog in there right so um so yeah
so essentially that that was what I
wanted to discuss today right so three
mistakes that we did that hopefully uh
you're not going to make right hopefully
this talk with Will resonate with a few
of you and um I want to really um
encourage you to reach out to me talk to
me like if you've ever done like a large
scale migration we're about to do a
large scale migration of our ado
um we've migrated just a few use case so
far but we're about to do like a huge
migration next month I really would like
if you if you have any any experience
with that reach out to me right share
your learnings I would love to hear
about it right and then um yeah so thank
you for assisting for this this talk and
um if you have any questions feel free
to ask
away
w
5.0 / 5 (0 votes)