Bell's data modernization journey: executing for the future with Google

Google Cloud

1 Jul 202413:03

Summary

TLDREric from Bell's Enterprise data platform team shares his journey from intern to architect and discusses the company's transition to Google Cloud. He highlights three key mistakes made during their on-premise data management: proliferating data silos, neglecting developer productivity, and poor data governance practices. Eric outlines how Bell is addressing these issues by standardizing on a single data platform, improving developer tools, and adopting data cataloging with Google Cloud's Dataplex.

Takeaways

📊 Bell is Canada's largest telecom provider, serving multiple provinces and regions.
🕹️ The speaker, Eric, works in Bell's Enterprise Data Platform team and has been with the company for 17 years, starting as an intern.
🏆 The speaker used his experience in competitive gaming to draw parallels between analyzing mistakes in games and reviewing past data management strategies.
☁️ Bell has recently migrated to Google Cloud and is reviewing past mistakes to avoid repeating them in the new environment.
🏗️ The first major mistake identified was the creation of numerous data silos, leading to inefficiencies and challenges in data management.
🚧 The second mistake was the lack of focus on improving developer productivity, leading to redundant and inefficient coding practices.
🛠️ Bell is now working on standardizing their data platform, consolidating their data into Google BigQuery, and improving data accessibility.
💻 The company is also implementing tools like Terraform modules and Backstage to standardize and streamline developer workflows.
🔄 Bell plans to create config-driven libraries to reduce redundant coding and improve the efficiency of data pipeline creation.
📑 The third mistake involved poor data governance practices, with metadata managed in Excel and Confluence. Bell is now standardizing on DataPlex for better data cataloging and governance.

Q & A

Who is the speaker, and what is his role at Bell?
-The speaker is Eric, who works in the Enterprise Data Platform team at Bell. He has been with the company for 17 years, starting as an intern and then moving into roles such as Data Engineer and currently, an Architect.
What is Bell, and what regions does it serve?
-Bell is the largest telecommunications provider in Canada, serving regions like Quebec, Ontario, Atlantic provinces, and Manitoba.
Why does the speaker mention playing competitive video games, and how does it relate to the talk?
-The speaker mentions playing competitive video games like Rocket League to draw a parallel between analyzing mistakes in gaming and analyzing mistakes made during Bell's data platform migration. The idea is that reviewing mistakes is crucial for improvement in both contexts.
What were the three major mistakes Bell made with their on-premise data systems?
-The three major mistakes were: 1) Deploying dozens of data environments and creating data silos. 2) Failing to improve developers' productivity and experience. 3) Poor data governance practices, including the management of metadata in Excel spreadsheets and Confluence.
What was the impact of deploying multiple data environments at Bell?
-Deploying multiple data environments led to a complex web of data pipelines, making it difficult for engineers to create value and requiring them to spend a lot of time moving data around instead of solving core problems.
How did Bell plan to address the issue of multiple data environments in their migration to Google Cloud?
-Bell planned to standardize on a single data platform in Google Cloud, specifically using Google BigQuery. This would centralize data and reduce the need for engineers to move data across different environments.
What problems did Bell encounter with developer productivity, and how are they addressing these in the cloud migration?
-Bell found that developers were repeatedly solving the same problems and writing redundant code, particularly in areas like data compaction. To address this, they are building a solid foundation with Terraform modules and using tools like Backstage to standardize and automate processes, improving productivity.
What is Backstage, and how is Bell utilizing it in their cloud migration?
-Backstage is an open-source developer portal created by Spotify that allows engineers to use templates to quickly spin up applications. Bell is using Backstage to simplify and standardize their development processes during their cloud migration.
What issue did Bell face with data governance, and what tool are they using to improve it?
-Bell struggled with managing table metadata using Excel spreadsheets and Confluence, leading to inefficiencies. To improve data governance, they are now standardizing with Google DataPlex to catalog assets and provide a centralized location for data governance.
What is the speaker's main message to the audience regarding their own experiences with large-scale migrations?
-The speaker encourages the audience to learn from Bell's mistakes and shares their experiences to help others avoid similar pitfalls. He invites others with experience in large-scale migrations to reach out and share their learnings, particularly as Bell prepares for a significant migration next month.

Outlines

00:00

🎤 Introduction and Background

Eric introduces himself as a member of the Enterprise Data Platform team at Bell, where he has worked for 17 years, starting as an intern. He gives a brief overview of Bell as Canada’s largest telecom provider, emphasizing its extensive service coverage. Eric then shares a personal anecdote about playing competitive video games like Rocket League, using it as a metaphor to discuss learning from mistakes, which ties into the main theme of his talk: reflecting on past errors in Bell’s data management and sharing three major mistakes made during their transition to Google Cloud.

05:01

🏗️ Mistake 1: Data Silos and Fragmented Systems

Eric delves into the first major mistake made at Bell: the proliferation of data environments and silos over the years. Initially, the goal was to create a unified data lake when implementing Hadoop in 2016. However, the effort was abandoned midway, leading to a fragmented system where engineers worked in isolated tech stacks, resulting in a complex web of data pipelines. This fragmentation hindered efficient data usage and slowed down processes like data monetization. The solution, Eric explains, is to standardize on a single data platform in Google Cloud, consolidating data to streamline access and reduce the need for redundant pipelines.

10:01

🚀 Mistake 2: Neglecting Developer Productivity

The second mistake Eric discusses is the failure to prioritize developer productivity on Bell’s on-premise systems. Developers were burdened with managing infrastructure and repeatedly solving the same problems due to a lack of standardized tools. For example, multiple teams independently developed similar libraries for tasks like data compaction in Hadoop. To address this, Bell is now focused on building a solid foundation in the cloud, starting with standardized Terraform modules and leveraging tools like Backstage, an open-source developer portal created by Spotify, to improve the efficiency of application development and reduce code redundancy.

🛠️ Mistake 3: Poor Data Governance Practices

The third and final mistake highlighted is Bell’s inadequate data governance practices. Eric admits that table metadata was being managed in Excel spreadsheets and Confluence, leading to inefficiencies and confusion among engineers. Moving to the cloud, Bell is now standardizing with Dataplex to properly catalog and manage data assets, ensuring that information is easily accessible and well-organized. This shift aims to eliminate the chaos of managing data across multiple platforms and improve overall data governance. Eric concludes by inviting the audience to share their experiences with large-scale migrations and offering to connect for further discussions.

Mindmap

Keywords

💡Data Silos

Data silos refer to isolated data storage systems or environments where data is kept separate from other data. In the video, the speaker mentions that Bell deployed dozens of data silos over the years, which led to inefficiencies and difficulties in managing and accessing data. These silos made it challenging to create value and implement technologies like AI/ML, as engineers spent more time managing data pipelines rather than solving core problems.

💡Cloud Migration

Cloud migration is the process of moving data, applications, and other business elements from on-premises environments to a cloud computing environment. The video discusses Bell's ongoing cloud migration to Google Cloud, emphasizing the opportunity to correct past mistakes made with on-premise systems, such as data silos and inefficient data governance practices. The migration aims to standardize and modernize Bell's data infrastructure.

💡Data Governance

Data governance involves the management and oversight of data assets to ensure data quality, consistency, and security across an organization. The speaker highlights poor data governance practices at Bell, such as managing metadata in Excel spreadsheets and Confluence, which led to inefficiencies. The migration to Google Cloud includes adopting better data governance tools like Dataplex to catalog and manage data more effectively.

💡Developer Productivity

Developer productivity refers to the efficiency and effectiveness of software engineers in creating and maintaining code. In the video, the speaker stresses that Bell did not initially focus on improving developer productivity, leading to redundant work and inefficiencies. By modernizing their approach in the cloud, including using tools like Backstage and standardizing processes with Terraform, Bell aims to significantly enhance developer productivity.

💡Terraform

Terraform is an open-source infrastructure as code software tool that enables users to define and provision data center infrastructure using a high-level configuration language. The video mentions that Bell is using Terraform to build standardized modules that enforce best practices, naming conventions, and other standards, helping to prevent the repetitive coding issues they experienced on-premise.

💡Backstage

Backstage is an open-source platform created by Spotify that helps developers manage and streamline their work by providing templates and tools in a single interface. The speaker mentions that Bell conducted a proof of concept with Backstage and found it promising for improving developer experience. Backstage will be used to allow developers to quickly spin up applications, reducing redundancy and increasing efficiency.

💡Data Pipelines

Data pipelines are a series of data processing steps that move data from one system to another. The video describes how Bell's legacy systems resulted in a 'spaghetti' of data pipelines, leading to inefficiencies and challenges in creating value. The cloud migration aims to simplify and centralize data pipelines, reducing the need for engineers to focus on data movement and allowing them to concentrate on more value-added tasks.

💡ETL (Extract, Transform, Load)

ETL stands for Extract, Transform, Load, a process in data warehousing that involves extracting data from various sources, transforming it into a suitable format, and loading it into a destination database. The speaker notes that Bell's ETL processes were only partially migrated to their data lake in 2016, contributing to the fragmented data environment. The cloud migration includes plans to improve and modernize ETL processes.

💡Data Monetization

Data monetization is the process of using data to generate revenue. In the video, the speaker mentions that Bell is heavily involved in data monetization, which has been hampered by the fragmented and inefficient data systems. By centralizing data in the cloud, Bell aims to streamline data access and analysis, thereby enhancing their ability to monetize data effectively.

💡Data Lake

A data lake is a storage system that holds vast amounts of raw data in its native format until it is needed. Bell attempted to create a data lake in 2016 with Hadoop, intending to consolidate their data for easier access and analysis. However, the initiative was only partially completed, leading to continued data silos. The cloud migration includes plans to complete this vision and create a more unified data environment.

Highlights

Introduction by Eric, who has worked at Bell for 17 years, starting as an intern and now an architect in the Enterprise Data Platform team.

Bell is the largest telecom provider in Canada, serving regions including Quebec, Ontario, Atlantic provinces, and Manitoba.

Eric's playful icebreaker about competitive video games, sharing his personal interest in Rocket League.

Bell's recent migration to Google Cloud, aiming to learn from past mistakes made on-premises.

Highlighting three major mistakes made by Bell in their data practices: deploying data silos, neglecting developer productivity, and poor data governance.

Mistake 1: Creating multiple data silos over time, leading to fragmented data environments and inefficient data management.

The shift to a standardized data platform on Google BigQuery to centralize data and streamline access.

Mistake 2: Neglecting developer productivity by focusing too much on infrastructure management rather than improving developer experience.

Efforts to modernize by building Terraform modules for standardized naming conventions, best practices, and labeling in the cloud.

Bell's proof of concept with Backstage, an open-source developer portal created by Spotify, to streamline application development.

Mistake 3: Poor data governance practices, including managing metadata in Excel spreadsheets and Confluence.

Adoption of Google DataPlex for standardized data governance, improving metadata management and reducing reliance on outdated methods.

Encouragement to the audience to avoid these common mistakes, emphasizing the importance of learning from Bell's experiences.

Eric's invitation to the audience to share their experiences with large-scale migrations, expressing a desire to learn from others.

Closing remarks with an open invitation for questions and further discussion on the topic.