Google SWE teaches systems design | EP22: HBase/BigTable Deep Dive
Summary
TLDRIn this video, the host celebrates the one-month anniversary of their channel and dives into a detailed exploration of HBase, a NoSQL database modeled after Google's Bigtable. They discuss HBase's architecture, including its master and region servers, and how it uses LSM trees and SS tables for efficient random read and write performance. The video compares HBase with Cassandra, highlighting differences in their use cases, and touches on HBase's strong consistency and integration with big data processing frameworks like MapReduce and Spark. The host concludes by differentiating HBase's suitability for data lakes and analytics from traditional data warehouses.
Takeaways
- 🎉 The speaker celebrated the one-month anniversary of their channel with 260 subscribers.
- 📚 The video discusses HBase, a NoSQL database modeled after Google's Bigtable.
- 🔍 The speaker had to research HBase deeply due to the lack of quality information online, with many blog posts plagiarizing each other.
- 🌐 HBase uses LSM trees and SS tables to achieve better random read and write performance with lower latency compared to HDFS.
- 📈 HBase is a wide column store NoSQL database, similar to Cassandra but with significant differences.
- 🔑 HBase organizes data with a single row key and column families, where the row key can be composed of multiple parts.
- 💾 The architecture of HBase includes a master server for metadata and operations, region servers for handling data, and uses ZooKeeper for coordination.
- 🚀 HBase is not ideal for time series data due to potential hotspots caused by sequential writes to a single partition.
- ♻️ Writes in HBase first go to an in-memory LSM tree and then are flushed to SS tables, which are stored in a column-oriented format on HDFS.
- 🔒 HBase provides strong consistency for data replication, ensuring all writes are successful before being considered complete.
- 📊 HBase integrates well with analytics engines like MapReduce, Spark, and Tez due to its column-oriented storage on HDFS.
- 🚀 HBase is suitable for use cases involving large-scale batch jobs and stream processing, especially in a data lake scenario.
Q & A
What is the significance of the one-month anniversary mentioned in the script?
-The one-month anniversary mentioned in the script signifies that the speaker's channel has reached a milestone of one month, and they are excited to have 260 subscribers, indicating growth and engagement with the audience.
What is HBase and why is it discussed in the script?
-HBase is a wide column NoSQL storage database that is discussed in the script due to its relevance to the topic of NoSQL databases and its use of LSM trees and SS tables for better performance compared to HDFS.
What are the key differences between HBase and Cassandra mentioned in the script?
-The script mentions that while both HBase and Cassandra are wide column NoSQL databases, they differ significantly in terms of their architecture, replication strategy, and suitability for different use cases such as real-time transactions and analytics processing.
How does HBase achieve better random read and write performance compared to HDFS?
-HBase uses LSM trees and SS tables to achieve better random read and write performance. This allows for lower latency reads and writes, as opposed to the high latency and sequential appends/truncates of HDFS.
What is the role of the master server in HBase?
-The master server in HBase is responsible for storing file metadata, handling operations on the metadata, and managing the location of files. It also performs range-based partitioning based on the row key and can split partitions if they become too large or have too much load.
What is the purpose of the region server in HBase?
-The region server in HBase handles the actual storage and retrieval of data. It runs on HDFS data node servers and maintains an in-memory LSM tree data structure for efficient writes. It also communicates with HDFS for data replication and storage of SS tables.
How does HBase integrate with HDFS for data replication?
-HBase integrates with HDFS by using the HDFS data node as the region server and leveraging the replication pipeline of HDFS to maintain the required replication factor for SS tables, ensuring data redundancy and fault tolerance.
What is the significance of column-oriented storage in HBase?
-Column-oriented storage in HBase allows for high read throughput when reading values over a column or partition, which is beneficial for analytics processing and batch jobs that require scanning large amounts of data.
How does HBase differ from a traditional SQL database in terms of data handling?
-HBase, being a NoSQL database, does not have structured data and does not support native joins like a SQL database. Instead, it requires batch processes or other methods to perform data associations and joins.
What is the advantage of using HBase for a data lake scenario?
-Using HBase for a data lake scenario allows for the dumping of unstructured data into a format that can later be used for analytics. HBase provides the advantage of good read and write performance for random access, which is useful for transaction processing in addition to analytics.
What are the limitations of HBase when it comes to real-time transaction processing?
-While HBase uses LSM trees for efficient writes, it may not handle writes as fast as Cassandra, which is designed for real-time transaction processing. HBase is more suited for scenarios where high read throughput for analytics is required.
Outlines
📅 One-Month Anniversary and Introduction to HBase
The speaker celebrates the one-month anniversary of their channel, expressing excitement over the 260 subscribers and encouraging continued growth. They introduce the topic of HBase, a NoSQL database, expressing frustration with the lack of in-depth information online and the prevalence of plagiarized content. The speaker decides to delve deeper into HBase, comparing it to Cassandra, another wide-column NoSQL database, and highlighting the differences despite their similar functionalities. They also mention that HBase uses LSM trees and SS tables to achieve better performance in random read and write operations, contrasting with the high latency of HDFS.
🔍 HBase Data Model and Architecture Overview
This paragraph delves into the data model of HBase, which is a wide column store with a single row key and any number of column values grouped into column families. The speaker discusses the importance of designing an effective row key to ensure proper data sorting. They provide an architectural overview of HBase, including the roles of the master server, region server, and the use of ZooKeeper for coordination. The master server manages file metadata and operations, while the region server handles data storage and retrieval, leveraging LSM trees for efficient writes and SS tables for sorted, column-oriented storage. The paragraph also touches on the replication process and the strong consistency provided by HBase due to its reliance on HDFS.
🔄 HBase Replication and Analytics Processing
The speaker explains the replication process in HBase, where writes are first stored in an in-memory LSM tree and then flushed to HFiles, which are SS tables. These HFiles are propagated to HDFS data nodes and replicated to meet the required replication factor, ensuring data durability and availability. The paragraph also discusses the advantages of storing SS tables on HDFS for high read throughput and integration with big data processing engines like MapReduce, Spark, and Tez. The speaker notes that HBase is not suitable for SQL queries or joins due to its non-SQL nature, but is ideal for batch processing and analytics on large datasets, making it a good fit for data lakes where unstructured data is stored for later analysis.
Mindmap
Keywords
💡HBase
💡Wide Column NoSQL Database
💡LSM Trees
💡SS Tables
💡Row Key
💡Column Family
💡Master Server
💡Region Server
💡Zookeeper
💡Replication
💡Data Lake
Highlights
The channel reached its one-month anniversary with 260 subscribers.
HBase is a NoSQL storage database modeled after Google's Bigtable.
HBase uses LSM trees and SS tables for better random read and write performance.
HBase is a wide column database with a row key and column families.
Row keys in HBase can be composed of multiple parts for sorting data.
HBase architecture includes a master server, region server, and replication with a Zookeeper instance.
The master server in HBase is similar to the HDFS Name Node, handling file metadata and operations.
HBase is not suitable for time series data with a timestamp as the row key due to potential hotspots.
Region servers in HBase run on HDFS data node servers and handle reads and writes.
Writes in HBase first go to an in-memory LSM tree structure before being flushed to SS tables.
SS tables in HBase are stored in a column-oriented format for high read throughput.
HBase integrates well with MapReduce and dataflow engines like Spark and Tez for analytics.
HBase does not support SQL queries or structured data with native joins.
HBase is suitable for data lakes, allowing transaction processing with the ability to run batch jobs.
Cassandra is better for real-time transaction processing with user-facing applications.
HBase is master-oriented with a focus on read throughput for analytics processing.
HBase provides advantages over HDFS for transaction processing with improved read and write performance.
The speaker plans to discuss key-value stores, such as Riak, Redis, Memcached, and caching in general, in upcoming videos.
Transcripts
all right um i'm back again uh welcome
everyone
i should just say yesterday was the one
month anniversary of this channel so i'm
pretty you know overwhelmed and excited
that we already have 260 people
subscribed so let's keep those numbers
up but uh today we're going to talk
about hbase it's kind of wild actually
because i was doing research for this
topic in order to make a video about it
and there's literally like probably 10
blog posts which are just blatant
plagiarisms of one another and just copy
the same like bullcrap information that
go into absolutely zero depth so i
actually had to dive into a little bit
here and then right as all that happens
it occurs to me that hbase is modeled
after bigtable which there's literally a
ton of information on so i'm an idiot
but anyways uh let's get into this video
all right so hbase what is it um well
the reason i'm talking about hbase in
the order that i'm talking about it in
is a couple things first of all it
allowed me to touch on hdfs in the prior
video but more importantly hbase is
another wide column nosql storage
database and the reason i want to talk
about this now is the last database i
talked about cassandra was also a wide
column nosql database however there are
some pretty serious differences between
the two even though on the surface level
they look like they serve similar
functionalities
so basically we're going to compare
those two but just to give an overview
even though hadoop or the the file
system provides pretty high latency
rights and reads since they're all
straight from disk
and only allows basically sequential
appends and truncates we'll see how
hbase uses lsm trees and ss tables in
order to achieve
better random read and write performance
in addition to lower latency reads and
writes performances
okay so in terms of the data model hbase
is a wide column database so it's not
identical to cassandra but it looks
pretty damn similar where each row has a
single row key and then basically any
column values that they want and there's
this concept of like a column family as
well which is kind of like you know
combining uh you know two values to
create a column
the row key also can and probably should
be comprised of multiple parts so as
opposed to having those clustering keys
in cassandra where that allows like an
internal sort order instead row keys
just have you know maybe say multiple uh
delineations inside of them and you have
to be pretty clever when developing this
row key to make sure that your data is
sorted the way that you want it
okay so just as an image to kind of give
you a sense of the architectural
overview
we have this master server a region
server and then you know kind of the
replication that's going on in the
background there's also a zookeeper
instance which i'll mention a little bit
as well but keep in mind that um you
know if you're ever talking about hbase
it's very much modeled after google's
bigtable which basically does the same
exact thing as hbase however it's built
on the google file system or now
actually the updated version of the
google file system called colossus
so what is the master server well if you
remember the hdfs name node it's
actually very similar to that but it's
kind of it's its own thing in hbase so
it's going to store all the file
metadata and you know all the operations
that occur on the metadata
such as like renaming files or anything
like that and also the corresponding
chunks of where all the files are
located
and so this is basically how hbase does
its partitioning
it's a range based partitioning on that
original row key
and if a given partition gets too big or
it has too much load
you can go ahead and split that up
the one thing to note here is that means
that hbase is not good for usage
patterns
say with like time series data where you
know the time stamp is the row key
because then every single write is going
to be going to one partition at a time
and it's going to create hotspots so in
that case you might want to use
something like a hash of a key
okay
in terms of reads and writes the client
is going to reach out to that hmaster
server in order to figure out the
location of the file and then you know
it's going to lead it to a region server
which i'll touch upon next and then
presumably in order to ensure high
availability the master server like the
master server and htfs can use that
zookeeper instance in order to keep a
eventually consistent write ahead log
and that way you can have a backup
master server reading from zookeeper in
order to eventually pull that right
ahead log and stay consistent with the
master server
okay what is a region server well a
region server before you guys kind of
get into the wrong mindset here a region
server tends to be run on hdfs data node
servers so as opposed to being its own
dedicated server usually it's just a
second program that's being run on those
data nodes in conjunction with whatever
the program is running that actually you
know makes a computer a data node so
what the region server does is it
occasionally sends heartbeats to
zookeeper so basically says okay i'm
alive you can still send rights to me or
reads to me
additionally on that actual region
server we're holding an in-memory lsm
tree style data structure and that's
where rights are going to first be set
so if you remember from literally my
first video on this channel we send
rights to the lsm tree
this is a pretty efficient write
structure because it means that they
first go to memory
and then reads once you want them first
go to the lsm tree and if the lsm tree
gets too big it gets flushed to ss
tables and then if whatever the key it
is that you want to read is not in the
lsm tree you look towards the ss tables
and start sorting through those and
there are a bunch of optimizations on
that that you can do in order to kind of
speed up that read process like bloom
filters
and you know other types of caching
so ss tables are actually going to be
stored in column oriented format so that
means that as opposed to just having
you know the the key and then the entire
value of the row what you have are these
sorted tables where
it's all the values of the column in
turn so that column oriented storage is
something that i've mentioned in my data
data warehousing video but basically
what it means that you can achieve
really high throughput over an entire
table if you say you just want one
column
okay so now let's talk about replication
because this is kind of the whole point
of being built on hdfs in addition to
kind of the analytical processing that
we'll talk about next
so like i said region nodes generally
speaking are going to be putting rights
in this mem store and then once that mem
store gets too big they're going to
flush them to an h file which is just an
ss table
and since the region node is generally
speaking on or near an hdfs data node
all it's going to do is once that hdfs
file so the h file is going to be
propagated to that data node the data
node is going to use the replication
pipeline of hdfs that we talked about in
the previous video on this channel in
order to go ahead and reach the required
replication factor of that ss table so
even though the region server is
effectively going to be handling all
rights and reads for a given
like partition of data it may
technically be communicating with one or
many of the replicas
that is going to be holding that ss
table for example if a given data node
goes down the region server is probably
going to have to be talking to a
different data node
um oh and furthermore uh we should just
keep in mind the fact that technically
that is strong consistency because um
everything has to kind of succeed in
hdfs for
the replication to be considered
successful and the right to be
considered successful in the first place
but okay in terms of the analytics
perspective what good does it do us
storing all of these ss tables on hdfs
as opposed to just you know
kind of having them on any normal
database
well for starters the fact that they are
column oriented storage means that we
can have really high read throughput
when trying to read over all the values
on one partition and a column
and additionally built being built on
hdfs means really good integration with
mapreduce and dataflow engines like
spark and tes
one thing to note is that since hbase is
not a sql database it doesn't have
structured data you can't actually
perform native joins what you have to do
is instead use something like a batch
process to perform joints and i talked
about those in the batch process video
basically you are trying to find data
associations perhaps even using two sets
of mappers and then combining those
together into one set of producers
okay so in conclusion even though hbase
looks really similar to cassandra on a
surface level they're very very
different in terms of what you want to
be doing with them cassandra's really
nice for real-time transactions
processing so that means that basically
anything user facing where you just want
like a really quick write so you can get
them a really quick success message and
then have that in the database to you
know have that eventual consistency
between all of these cassandra instances
that's good that's kind of like this
leaderless replication strategy it's
very um based off dynamo hbase on the
other hand is a master
you know master oriented database in
terms of the replication strategy and so
even though they both use lsm trees
hbase isn't going to be able to handle
those rights as fast
however in terms of reads and analytics
processing
you can achieve very high read
throughput over a column which is super
useful if you want to run a huge batch
job or a stream job
this is great um for something called
being a data lake which is basically
saying i'm going to dump a ton of
unstructured data in kind of like a
database format into one place and then
eventually run analytics later it's just
that the advantage of doing this with
hbase over hdfs itself is that you can
get
nice read and write performance for both
random reads and writes in the event
that you do have to do some general like
you know transaction processing so it's
basically a data lake that allows you to
do some transaction processing but keep
in mind that data lakes also are not
exactly the same as data warehouses
which are specifically for structured
data with a ton of data associations if
you recall data warehouses use that
whole like stars and snowflake schema
where you have a huge fact table which
is very structured and then a ton of
dimension and sub-dimension tables which
reference the fact table or actually the
fact table is going to reference those
dimension tables
but the point is if you just want to be
able to run sql queries here hbase is
not the thing but if you want to be able
to dump in a ton of data and then
eventually run you know a ton of
expensive batch jobs on it then hbase is
really nice for you
so yeah all right guys i think uh next
i'm going to start moving into key value
stores so i can talk about things like
ryak redis memcached and just caching in
general all right have a good
Посмотреть больше похожих видео
Choosing a Database for Systems Design: All you need to know in one video
Google SWE teaches systems design | EP1: Database Design
Introduction to NoSQL databases
How do NoSQL databases work? Simply Explained!
Google's Tech Stack (6 internal tools revealed)
How databases scale writes: The power of the log ✍️🗒️
5.0 / 5 (0 votes)