Google SWE teaches systems design | EP22: HBase/BigTable Deep Dive

Summary

In this video, the host celebrates the one-month anniversary of their channel and dives into a detailed exploration of HBase, a NoSQL database modeled after Google's Bigtable. They discuss HBase's architecture, including its master and region servers, and how it uses LSM trees and SS tables for efficient random read and write performance. The video compares HBase with Cassandra, highlighting differences in their use cases, and touches on HBase's strong consistency and integration with big data processing frameworks like MapReduce and Spark. The host concludes by differentiating HBase's suitability for data lakes and analytics from traditional data warehouses.

Takeaways

• 🎉 The speaker celebrated the one-month anniversary of their channel with 260 subscribers.
• 📚 The video discusses HBase, a NoSQL database modeled after Google's Bigtable.
• 🔍 The speaker had to research HBase deeply due to the lack of quality information online, with many blog posts plagiarizing each other.
• 🌐 HBase uses LSM trees and SS tables to achieve better random read and write performance with lower latency compared to HDFS.
• 📈 HBase is a wide column store NoSQL database, similar to Cassandra but with significant differences.
• 🔑 HBase organizes data with a single row key and column families, where the row key can be composed of multiple parts.
• 💾 The architecture of HBase includes a master server for metadata and operations, region servers for handling data, and uses ZooKeeper for coordination.
• 🚀 HBase is not ideal for time series data due to potential hotspots caused by sequential writes to a single partition.
• ♻️ Writes in HBase first go to an in-memory LSM tree and then are flushed to SS tables, which are stored in a column-oriented format on HDFS.
• 🔒 HBase provides strong consistency for data replication, ensuring all writes are successful before being considered complete.
• 📊 HBase integrates well with analytics engines like MapReduce, Spark, and Tez due to its column-oriented storage on HDFS.
• 🚀 HBase is suitable for use cases involving large-scale batch jobs and stream processing, especially in a data lake scenario.

Q & A

What is the significance of the one-month anniversary mentioned in the script?

The one-month anniversary mentioned in the script signifies that the speaker's channel has reached a milestone of one month, and they are excited to have 260 subscribers, indicating growth and engagement with the audience.

What is HBase and why is it discussed in the script?

HBase is a wide column NoSQL storage database that is discussed in the script due to its relevance to the topic of NoSQL databases and its use of LSM trees and SS tables for better performance compared to HDFS.

What are the key differences between HBase and Cassandra mentioned in the script?

The script mentions that while both HBase and Cassandra are wide column NoSQL databases, they differ significantly in terms of their architecture, replication strategy, and suitability for different use cases such as real-time transactions and analytics processing.

How does HBase achieve better random read and write performance compared to HDFS?

HBase uses LSM trees and SS tables to achieve better random read and write performance. This allows for lower latency reads and writes, as opposed to the high latency and sequential appends/truncates of HDFS.

What is the role of the master server in HBase?

The master server in HBase is responsible for storing file metadata, handling operations on the metadata, and managing the location of files. It also performs range-based partitioning based on the row key and can split partitions if they become too large or have too much load.

What is the purpose of the region server in HBase?

The region server in HBase handles the actual storage and retrieval of data. It runs on HDFS data node servers and maintains an in-memory LSM tree data structure for efficient writes. It also communicates with HDFS for data replication and storage of SS tables.

How does HBase integrate with HDFS for data replication?

HBase integrates with HDFS by using the HDFS data node as the region server and leveraging the replication pipeline of HDFS to maintain the required replication factor for SS tables, ensuring data redundancy and fault tolerance.

What is the significance of column-oriented storage in HBase?

Column-oriented storage in HBase allows for high read throughput when reading values over a column or partition, which is beneficial for analytics processing and batch jobs that require scanning large amounts of data.

How does HBase differ from a traditional SQL database in terms of data handling?

HBase, being a NoSQL database, does not have structured data and does not support native joins like a SQL database. Instead, it requires batch processes or other methods to perform data associations and joins.

What is the advantage of using HBase for a data lake scenario?

Using HBase for a data lake scenario allows for the dumping of unstructured data into a format that can later be used for analytics. HBase provides the advantage of good read and write performance for random access, which is useful for transaction processing in addition to analytics.

What are the limitations of HBase when it comes to real-time transaction processing?

While HBase uses LSM trees for efficient writes, it may not handle writes as fast as Cassandra, which is designed for real-time transaction processing. HBase is more suited for scenarios where high read throughput for analytics is required.

Outlines

00:00

📅 One-Month Anniversary and Introduction to HBase

The speaker celebrates the one-month anniversary of their channel, expressing excitement over the 260 subscribers and encouraging continued growth. They introduce the topic of HBase, a NoSQL database, expressing frustration with the lack of in-depth information online and the prevalence of plagiarized content. The speaker decides to delve deeper into HBase, comparing it to Cassandra, another wide-column NoSQL database, and highlighting the differences despite their similar functionalities. They also mention that HBase uses LSM trees and SS tables to achieve better performance in random read and write operations, contrasting with the high latency of HDFS.

05:01

🔍 HBase Data Model and Architecture Overview

This paragraph delves into the data model of HBase, which is a wide column store with a single row key and any number of column values grouped into column families. The speaker discusses the importance of designing an effective row key to ensure proper data sorting. They provide an architectural overview of HBase, including the roles of the master server, region server, and the use of ZooKeeper for coordination. The master server manages file metadata and operations, while the region server handles data storage and retrieval, leveraging LSM trees for efficient writes and SS tables for sorted, column-oriented storage. The paragraph also touches on the replication process and the strong consistency provided by HBase due to its reliance on HDFS.

10:02

🔄 HBase Replication and Analytics Processing

The speaker explains the replication process in HBase, where writes are first stored in an in-memory LSM tree and then flushed to HFiles, which are SS tables. These HFiles are propagated to HDFS data nodes and replicated to meet the required replication factor, ensuring data durability and availability. The paragraph also discusses the advantages of storing SS tables on HDFS for high read throughput and integration with big data processing engines like MapReduce, Spark, and Tez. The speaker notes that HBase is not suitable for SQL queries or joins due to its non-SQL nature, but is ideal for batch processing and analytics on large datasets, making it a good fit for data lakes where unstructured data is stored for later analysis.

Keywords

💡HBase

HBase is an open-source, distributed, versioned, non-relational database modeled after Google's Bigtable. It is designed to provide real-time read/write access to large amounts of structured data. In the video, HBase is discussed as a wide column NoSQL storage database that uses LSM trees and SS tables for better performance compared to HDFS, and is compared with Cassandra in terms of functionality and use cases.

💡Wide Column NoSQL Database

A wide column NoSQL database is a type of non-relational database that stores data in tables with a dynamic number of columns. The script mentions that both HBase and Cassandra are wide column NoSQL databases, but they differ significantly in their architecture and use cases, which is a central theme of the video.

💡LSM Trees

LSM Trees, or Log-Structured Merge-Trees, are a type of data structure used in database systems to optimize write operations. The video explains that HBase uses LSM trees to achieve better random read and write performance with lower latency compared to the high latency writes and reads provided by HDFS.

💡SS Tables

SS Tables, or Sorted String Tables, are a part of the LSM tree data structure and are used for storing data on disk. The script discusses how HBase uses SS tables in conjunction with LSM trees to handle data that has been flushed from memory, allowing for efficient read operations.

💡Row Key

In the context of HBase, the row key is a unique identifier for each row in a table. The video script explains that the row key can be comprised of multiple parts and is crucial for determining the sorting and partitioning of data within the database.

💡Column Family

A column family is a group of columns in a wide column store that are often accessed together. The script mentions column families in the context of HBase's data model, where each row has a row key and can have any number of column values grouped into families.

💡Master Server

The master server in HBase is responsible for managing the metadata of the file system and coordinating operations such as file renaming. The video script describes the role of the master server in HBase's architecture, comparing it to the NameNode in HDFS.

💡Region Server

A region server in HBase is responsible for serving a particular range of data. The script explains that region servers are usually run on HDFS data node servers and handle the in-memory LSM tree data structure for efficient read and write operations.

💡Zookeeper

Zookeeper is a coordination service used for maintaining configuration information, naming, providing distributed synchronization, and providing group services. In the video, Zookeeper is mentioned as being used by HBase for maintaining an eventually consistent write-ahead log and for region server heartbeats.

💡Replication

Replication in the context of HBase refers to the process of copying data to multiple nodes to ensure data availability and fault tolerance. The script discusses how HBase leverages HDFS's replication mechanism to maintain the required replication factor for SS tables.

💡Data Lake

A data lake is a storage repository that holds a vast amount of raw data in its native format until it is needed. The video script mentions data lakes in the context of using HBase for storing unstructured data and running analytics on it later, contrasting it with the structured approach of data warehouses.

Highlights

The channel reached its one-month anniversary with 260 subscribers.

HBase is a NoSQL storage database modeled after Google's Bigtable.

HBase uses LSM trees and SS tables for better random read and write performance.

HBase is a wide column database with a row key and column families.

Row keys in HBase can be composed of multiple parts for sorting data.

HBase architecture includes a master server, region server, and replication with a Zookeeper instance.

The master server in HBase is similar to the HDFS Name Node, handling file metadata and operations.

HBase is not suitable for time series data with a timestamp as the row key due to potential hotspots.

Region servers in HBase run on HDFS data node servers and handle reads and writes.

Writes in HBase first go to an in-memory LSM tree structure before being flushed to SS tables.

SS tables in HBase are stored in a column-oriented format for high read throughput.

HBase integrates well with MapReduce and dataflow engines like Spark and Tez for analytics.

HBase does not support SQL queries or structured data with native joins.

HBase is suitable for data lakes, allowing transaction processing with the ability to run batch jobs.

Cassandra is better for real-time transaction processing with user-facing applications.

HBase is master-oriented with a focus on read throughput for analytics processing.

HBase provides advantages over HDFS for transaction processing with improved read and write performance.

The speaker plans to discuss key-value stores, such as Riak, Redis, Memcached, and caching in general, in upcoming videos.

Transcripts

all right um i'm back again uh welcome

everyone

i should just say yesterday was the one

month anniversary of this channel so i'm

pretty you know overwhelmed and excited

that we already have 260 people

subscribed so let's keep those numbers

up but uh today we're going to talk

about hbase it's kind of wild actually

because i was doing research for this

topic in order to make a video about it

and there's literally like probably 10

blog posts which are just blatant

plagiarisms of one another and just copy

the same like bullcrap information that

go into absolutely zero depth so i

actually had to dive into a little bit

here and then right as all that happens

it occurs to me that hbase is modeled

after bigtable which there's literally a

ton of information on so i'm an idiot

but anyways uh let's get into this video

all right so hbase what is it um well

the reason i'm talking about hbase in

the order that i'm talking about it in

is a couple things first of all it

allowed me to touch on hdfs in the prior

video but more importantly hbase is

another wide column nosql storage

database and the reason i want to talk

about this now is the last database i

talked about cassandra was also a wide

column nosql database however there are

some pretty serious differences between

the two even though on the surface level

they look like they serve similar

functionalities

so basically we're going to compare

those two but just to give an overview

even though hadoop or the the file

system provides pretty high latency

rights and reads since they're all

straight from disk

and only allows basically sequential

appends and truncates we'll see how

hbase uses lsm trees and ss tables in

order to achieve

better random read and write performance

in addition to lower latency reads and

writes performances

okay so in terms of the data model hbase

is a wide column database so it's not

identical to cassandra but it looks

pretty damn similar where each row has a

single row key and then basically any

column values that they want and there's

this concept of like a column family as

well which is kind of like you know

combining uh you know two values to

create a column

the row key also can and probably should

be comprised of multiple parts so as

opposed to having those clustering keys

in cassandra where that allows like an

internal sort order instead row keys

just have you know maybe say multiple uh

delineations inside of them and you have

to be pretty clever when developing this

row key to make sure that your data is

sorted the way that you want it

okay so just as an image to kind of give

you a sense of the architectural

overview

we have this master server a region

server and then you know kind of the

replication that's going on in the

background there's also a zookeeper

instance which i'll mention a little bit

as well but keep in mind that um you

know if you're ever talking about hbase

it's very much modeled after google's

bigtable which basically does the same

exact thing as hbase however it's built

on the google file system or now

actually the updated version of the

google file system called colossus

so what is the master server well if you

remember the hdfs name node it's

actually very similar to that but it's

kind of it's its own thing in hbase so

it's going to store all the file

metadata and you know all the operations

that occur on the metadata

such as like renaming files or anything

like that and also the corresponding

chunks of where all the files are

located

and so this is basically how hbase does

its partitioning

it's a range based partitioning on that

original row key

and if a given partition gets too big or

it has too much load

you can go ahead and split that up

the one thing to note here is that means

that hbase is not good for usage

patterns

say with like time series data where you

know the time stamp is the row key

because then every single write is going

to be going to one partition at a time

and it's going to create hotspots so in

that case you might want to use

something like a hash of a key

okay

in terms of reads and writes the client

is going to reach out to that hmaster

server in order to figure out the

location of the file and then you know

it's going to lead it to a region server

which i'll touch upon next and then

presumably in order to ensure high

availability the master server like the

master server and htfs can use that

zookeeper instance in order to keep a

eventually consistent write ahead log

and that way you can have a backup

master server reading from zookeeper in

order to eventually pull that right

ahead log and stay consistent with the

master server

okay what is a region server well a

region server before you guys kind of

get into the wrong mindset here a region

server tends to be run on hdfs data node

servers so as opposed to being its own

dedicated server usually it's just a

second program that's being run on those

data nodes in conjunction with whatever

the program is running that actually you

know makes a computer a data node so

what the region server does is it

occasionally sends heartbeats to

zookeeper so basically says okay i'm

alive you can still send rights to me or

reads to me

additionally on that actual region

server we're holding an in-memory lsm

tree style data structure and that's

where rights are going to first be set

so if you remember from literally my

first video on this channel we send

rights to the lsm tree

this is a pretty efficient write

structure because it means that they

first go to memory

and then reads once you want them first

go to the lsm tree and if the lsm tree

gets too big it gets flushed to ss

tables and then if whatever the key it

is that you want to read is not in the

lsm tree you look towards the ss tables

and start sorting through those and

there are a bunch of optimizations on

that that you can do in order to kind of

speed up that read process like bloom

filters

and you know other types of caching

so ss tables are actually going to be

stored in column oriented format so that

means that as opposed to just having

you know the the key and then the entire

value of the row what you have are these

sorted tables where

it's all the values of the column in

turn so that column oriented storage is

something that i've mentioned in my data

data warehousing video but basically

what it means that you can achieve

really high throughput over an entire

table if you say you just want one

column

okay so now let's talk about replication

because this is kind of the whole point

of being built on hdfs in addition to

kind of the analytical processing that

we'll talk about next

so like i said region nodes generally

speaking are going to be putting rights

in this mem store and then once that mem

store gets too big they're going to

flush them to an h file which is just an

ss table

and since the region node is generally

speaking on or near an hdfs data node

all it's going to do is once that hdfs

file so the h file is going to be

propagated to that data node the data

node is going to use the replication

pipeline of hdfs that we talked about in

the previous video on this channel in

order to go ahead and reach the required

replication factor of that ss table so

even though the region server is

effectively going to be handling all

rights and reads for a given

like partition of data it may

technically be communicating with one or

many of the replicas

that is going to be holding that ss

table for example if a given data node

goes down the region server is probably

going to have to be talking to a

different data node

um oh and furthermore uh we should just

keep in mind the fact that technically

that is strong consistency because um

everything has to kind of succeed in

hdfs for

the replication to be considered

successful and the right to be

considered successful in the first place

but okay in terms of the analytics

perspective what good does it do us

storing all of these ss tables on hdfs

as opposed to just you know

kind of having them on any normal

database

well for starters the fact that they are

column oriented storage means that we

can have really high read throughput

when trying to read over all the values

on one partition and a column

and additionally built being built on

hdfs means really good integration with

mapreduce and dataflow engines like

spark and tes

one thing to note is that since hbase is

not a sql database it doesn't have

structured data you can't actually

perform native joins what you have to do

is instead use something like a batch

process to perform joints and i talked

about those in the batch process video

basically you are trying to find data

associations perhaps even using two sets

of mappers and then combining those

together into one set of producers

okay so in conclusion even though hbase

looks really similar to cassandra on a

surface level they're very very

different in terms of what you want to

be doing with them cassandra's really

nice for real-time transactions

processing so that means that basically

anything user facing where you just want

like a really quick write so you can get

them a really quick success message and

then have that in the database to you

know have that eventual consistency

between all of these cassandra instances

that's good that's kind of like this

leaderless replication strategy it's

very um based off dynamo hbase on the

other hand is a master

you know master oriented database in

terms of the replication strategy and so

even though they both use lsm trees

hbase isn't going to be able to handle

those rights as fast

however in terms of reads and analytics

processing

you can achieve very high read

throughput over a column which is super

useful if you want to run a huge batch

job or a stream job

this is great um for something called

being a data lake which is basically

saying i'm going to dump a ton of

unstructured data in kind of like a

database format into one place and then

eventually run analytics later it's just

that the advantage of doing this with

hbase over hdfs itself is that you can

get

nice read and write performance for both

random reads and writes in the event

that you do have to do some general like

you know transaction processing so it's

basically a data lake that allows you to

do some transaction processing but keep

in mind that data lakes also are not

exactly the same as data warehouses

which are specifically for structured

data with a ton of data associations if

you recall data warehouses use that

whole like stars and snowflake schema

where you have a huge fact table which

is very structured and then a ton of

dimension and sub-dimension tables which

reference the fact table or actually the

fact table is going to reference those

dimension tables

but the point is if you just want to be

able to run sql queries here hbase is

not the thing but if you want to be able

to dump in a ton of data and then

eventually run you know a ton of

expensive batch jobs on it then hbase is

really nice for you

so yeah all right guys i think uh next

i'm going to start moving into key value

stores so i can talk about things like

ryak redis memcached and just caching in

general all right have a good