15 futuristic databases you’ve never heard of

Summary

The video script explores the evolution of databases, highlighting the shift from traditional SQL systems like MySQL and Postgres to modern, innovative databases. It reviews 15 emerging databases, including PlanetScale, CockroachDB, Neon, and others, focusing on their unique features such as serverless scalability, graph-based data models, and AI integration. The video emphasizes how these new technologies aim to solve the limitations of older systems, while also addressing challenges like cloud scaling, branching, and real-time analytics. It's a deep dive into the future of data management for developers.

Takeaways

• 🚀 The video discusses the need for speed and innovation in the tech industry, often symbolized by the phrase 'move fast and break things'.
• 🗄️ Relational Database Management Systems (RDBMS) like SQL have been foundational in computing, with Oracle and MySQL leading in their respective sectors.
• 🌐 The script introduces several new databases designed to handle modern challenges, including scalability and cloud-native operations.
• 🌌 PlanetScale is highlighted as a serverless platform for MySQL, leveraging Vitess for horizontal scaling, and offering a managed service with a user-friendly dashboard.
• 🧘‍♂️ Yugabyte offers a serverless database for PostgreSQL, aiming for infinite scalability and supporting multi-cloud strategies to avoid vendor lock-in.
• 🌟 Neon, currently in technical preview, is a serverless PostgreSQL option that can scale to zero and is written in Rust, offering a polished admin interface.
• 🔄 Dolt, inspired by Git, allows for database branching and merging, making it easier to experiment with data changes without disrupting the main database.
• 🦀 CockroachDB is a 'new SQL' database compatible with PostgreSQL, designed for horizontal scaling in the cloud, with a free tier and pay-as-you-go pricing.
• 💡 Cloudflare's D1 is a SQL Lite database for edge computing, allowing developers to write stored procedures in JavaScript for transaction management.
• 📊 Zeta is a serverless relational database with a spreadsheet-like interface, offering full-text search and branching capabilities, similar to Dolt.
• 📈 Eightbase is a relational database with a GraphQL API and a low-code tool for building front-end applications, aiming to simplify full-stack development.

Q & A

What is the concept behind 'move fast and break things' in the tech industry?

'Move fast and break things' is a philosophy encouraging rapid development and innovation, even if it involves taking risks or making mistakes along the way. It emphasizes speed over perfection, especially in early stages of development.

What is the relational database management system (RDBMS), and why is it important?

The relational database management system (RDBMS) organizes data in a structured way using rows and columns, forming relationships between different datasets. It's one of the most successful ideas in computing, still forming the backbone of many modern data systems like Oracle and MySQL.

Why is SQL considered outdated by some despite its widespread use?

SQL, developed in the 1970s, is considered outdated by some due to newer, more scalable database solutions and modern demands for handling big data. However, it remains critical as it powers many of today’s enterprise and web applications.

What problem did YouTube face with MySQL, and how was it solved?

YouTube faced scaling challenges with MySQL due to its massive traffic. Instead of switching to a NoSQL database, YouTube developed a library called Vitas, which allowed MySQL to scale horizontally, addressing the issue without changing the database system.

What are serverless databases, and how do platforms like PlanetScale and Neon use them?

Serverless databases are databases that automatically scale and manage infrastructure without user intervention. PlanetScale offers a MySQL-compatible serverless platform, and Neon provides a serverless PostgreSQL solution, with features like scaling to zero and advanced management dashboards.

What is 'branching' in databases, and why is it useful?

Branching in databases allows users to create experimental branches from the main database, similar to version control systems like Git. This enables developers to test changes without affecting the main database and merge successful changes later.

What is 'NewSQL' and how does CockroachDB fit into this category?

NewSQL refers to a class of databases that offer the scalability of NoSQL while maintaining the ACID properties and relational features of traditional SQL. CockroachDB is an example of NewSQL, built to scale horizontally while remaining compatible with PostgreSQL.

How does Cloudflare D1 differ from traditional SQL databases?

Cloudflare D1 is a lightweight, serverless SQL database running on the edge, designed for faster, localized queries. It offers stored procedures in JavaScript and can scale for web applications, though it lacks native transactions typical in SQL databases.

What are some advantages of using Zeta and 8base for database management?

Zeta provides a serverless relational database with a spreadsheet-like interface and full-text search capabilities. 8base, on the other hand, offers a relational database with a built-in GraphQL API and low-code tools for quickly building front-end applications.

How does SurrealDB combine features from different types of databases?

SurrealDB is a multi-model database that combines relational, document, and graph database features, offering flexibility in data modeling. It allows for SQL-like queries while supporting more complex, graph-based relationships, simplifying data interactions.

Outlines

00:00

🚀 Tech Success and the Evolution of Databases

This paragraph begins by humorously recounting a mishap in a tech job where moving fast led to a significant mistake—deleting a production database. The narrative transitions to discussing relational databases, specifically SQL, and their continued dominance in the tech world. SQL, developed in the 1970s, powers many modern platforms like YouTube and Facebook. The rise of new, cutting-edge databases that aim to 'move fast and break things' is highlighted, showcasing how innovation is reshaping traditional database management. The paragraph sets the stage for a deep dive into new databases attempting to improve or replace SQL.

05:02

🌐 PlanetScale and MySQL Scalability

This section introduces PlanetScale, a serverless platform based on MySQL and the open-source Vitas library. It explores how YouTube scaled MySQL using Vitas to handle millions of simultaneous connections, allowing for horizontal scaling. PlanetScale offers this technology as a fully managed service with an accessible dashboard and SDKs for integration. The paragraph also compares PlanetScale's pricing favorably to other cloud services like AWS and Google Cloud. It briefly mentions competing services like YogaByte and Neon, which aim to scale Postgres with features like multi-cloud hosting and serverless architecture.

🛠️ Dolt, CockroachDB, and Other New SQL Databases

Here, the focus shifts to Dolt, a database that combines MySQL with Git-like features, allowing users to branch, experiment, and merge changes easily. CockroachDB is also introduced, a 'New SQL' database built by ex-Google engineers for horizontal cloud scalability. Other alternatives such as FoundationDB, TitaniumDB, and Cloudflare's D1 are mentioned. D1, a lightweight SQL Lite database that runs on the edge, is particularly noted for allowing stored procedures to be written in JavaScript, simplifying transactions. The paragraph underscores the variety of new SQL databases designed to tackle modern scalability challenges.

📊 Zeta and Low-Code Database Solutions

The spotlight now turns to Zeta, another serverless relational database that integrates Postgres and Elasticsearch. Zeta stands out for treating data like a spreadsheet, offering developer-friendly tools with built-in full-text search capabilities. The paragraph also introduces 8base, a relational database that provides a GraphQL API and low-code tools for quickly building front-end apps. These tools position Zeta and 8base as developer-friendly alternatives to services like Airtable or Notion, emphasizing the growing trend of merging relational databases with ease-of-use and powerful search functionalities.

🌟 EdgeDB and the Post-SQL Era

This paragraph discusses EdgeDB, a graph-relational database powered by Postgres. Unlike traditional relational databases that rely on columns and rows, EdgeDB allows developers to define types (similar to programming languages) that naturally link to other types. This approach eliminates the need for complex joins and simplifies querying. It positions EdgeDB as more human-friendly and intuitive, describing it as an ORM-like solution for Postgres. The paragraph also briefly mentions SurrealDB, a graph-relational database with similar goals, highlighting its ease of use and scalability with Rust-powered performance.

🔄 Multi-Model Databases: SurrealDB and Fauna

This section expands on SurrealDB, which combines relational, document, and graph database elements to create an easy-to-use API. It offers horizontal scalability and ACID transaction support, positioning it as a flexible solution for complex data models. Fauna is another multi-model database created by ex-Twitter engineers, described as a document database with native joins. This feature makes it a strong choice for relational data management while maintaining the simplicity of document databases. The paragraph emphasizes Fauna's closed-source nature but highlights its ease of use and powerful capabilities.

⚡ Real-Time Databases: Memgraph, KeyDB, and Meilisearch

The focus here is on Memgraph, a real-time graph database similar to Neo4j but optimized for performance with C++ and memory-first architecture. Memgraph is ideal for real-time analytics platforms. KeyDB, a faster alternative to Redis, is also introduced, emphasizing its role as an in-memory cache for speeding up data retrieval in high-performance applications like Snap. Meilisearch, an open-source Rust-based full-text search engine, rounds out the section. It's noted for its ease of implementation, allowing developers to fine-tune search parameters and integrate search features quickly into web applications.

🤖 AI and Machine Learning Databases: MindsDB

The final paragraph introduces MindsDB, a database that integrates machine learning models directly into the database. Its 'AI Tables' allow users to train machine learning models using SQL-like commands and then make predictions from the data. This innovation simplifies the machine learning lifecycle by embedding model training and inference within the database itself. MindsDB is positioned as a forward-looking tool that could pave the way for more databases with built-in AI capabilities, offering a glimpse into the future of data management and machine learning.

Keywords

💡Relational Database Management System (RDBMS)

RDBMS is a database management system based on the relational model, which organizes data into tables with rows and columns. It is a foundational concept in database technology, as highlighted by the script's reference to SQL, which is a standard language for managing relational databases. The video discusses how traditional RDBMS solutions like Oracle and MySQL are being challenged by newer technologies, emphasizing the evolution in database management.

💡SQL

SQL, or Structured Query Language, is a programming language used to manage and manipulate relational databases. It is central to the video's theme as it is described as the backbone of modern Big Data systems. The script mentions that while SQL has been around since the 1970s, there is a new wave of databases aiming to innovate and potentially 'break things' in the traditional sense of how SQL databases operate.

💡Serverless

Serverless computing refers to a cloud computing execution model where the cloud provider dynamically manages the allocation and provisioning of servers. In the context of the video, 'serverless' databases like PlanetScale and YugaByte are discussed as innovative solutions that abstract away the server management, allowing for easier scaling and maintenance, which is a significant shift from traditional database management.

💡NoSQL

NoSQL databases are non-relational databases that differ from traditional RDBMS in that they do not use SQL for database operations. The video script contrasts NoSQL with SQL-based systems, suggesting that while NoSQL databases can offer scalability and flexibility, there is still significant innovation happening within the SQL space, as seen with new databases like CockroachDB.

💡Horizontal Scaling

Horizontal scaling, also known as 'outward scaling,' involves adding more machines or nodes to a system to increase its capacity. The video script mentions this concept in relation to databases like PlanetScale and CockroachDB, which are designed to scale out by distributing data across multiple servers, a critical feature for handling large volumes of data and high traffic loads.

💡PostgreSQL

PostgreSQL, often referred to as 'Postgres,' is a powerful, open-source object-relational database system. It is highlighted in the script as the underlying technology for several innovative databases, including YugaByte and Neon. The video emphasizes how Postgres serves as a robust foundation for new database solutions aiming to enhance traditional RDBMS capabilities.

💡Dolt

Dolt is a version control system for databases, allowing developers to manage database changes using familiar Git-like commands. The script introduces Dolt as a database that enables branching and merging, which is a novel approach to database version control, making it easier to experiment with data changes without affecting the main database.

💡GraphQL

GraphQL is a query language for APIs and a runtime for executing those queries with your existing data. The video mentions EightBase, a database that provides a GraphQL API for data, which simplifies how developers interact with databases by allowing them to request exactly the data they need, potentially reducing over-fetching and improving application performance.

💡Graph Database

A graph database is a type of database that uses graph structures with nodes, edges, and properties to represent and store data. The video script discusses EdgeDB and SurrealDB as examples of graph databases that are designed to handle complex relationships and queries more intuitively than traditional relational databases, which is particularly useful for applications involving interconnected data.

💡Multi-Model Database

A multi-model database supports multiple data models, such as document, key-value, graph, and column-family models, in one system. FaunaDB, mentioned in the script, is described as a multi-model database that combines the ease of use of a document database with the relational capabilities of SQL databases, making it a versatile solution for various data storage and retrieval needs.

💡AI Tables

AI Tables, as introduced by MindsDB in the video, is a concept that integrates machine learning models directly into the database. This allows for the creation of predictive models using SQL-like statements, which can then be used to make predictions. The video positions MindsDB as a forward-thinking database that simplifies the machine learning operations lifecycle by embedding AI capabilities within the database itself.

Highlights

The necessity to move fast and innovate in tech, as illustrated by the speaker's anecdote about their experience with a production database.

Relational database management systems (RDBMS) are a cornerstone of modern computing, with SQL remaining foundational despite being from the 1970s.

Oracle's dominance in the enterprise sector and MySQL's widespread use, particularly by platforms like YouTube and Facebook.

Introduction to PlanetScale, a serverless platform for MySQL that scales horizontally using the Vitess library.

YugaByte's open-source solution for PostgreSQL, aiming for infinite scalability in the cloud while maintaining native features.

Neon's serverless approach to PostgreSQL, featuring the ability to scale down to zero and a polished admin dashboard.

Dolt's innovative database branching inspired by Git, allowing for experimentation without disrupting the main database.

CockroachDB, developed by ex-Google employees, is a newSQL database designed for cloud scalability and compatibility with PostgreSQL.

FoundationDB and TitaniumDB as alternatives in the newSQL class, offering different approaches to scalability and performance.

Cloudflare's D1, a SQL Lite database for edge computing, allowing for data creation and querying on the edge.

Zeta, a serverless relational database with a unique spreadsheet-like approach to data management.

Eightbase's combination of a relational database with a GraphQL API and a low-code tool for building front-end applications.

EdgeDB, a graph-relational database that simplifies data relationships and queries by eliminating the need for joins.

SurrealDB's unique approach to data modeling, combining the strengths of SQL and graph databases with Rust's performance.

FaunaDB's multi-model capabilities, offering the ease of use of a document database with the power of native joins.

Memgraph, a real-time graph database designed for high-performance data pipelines, as an alternative to Neo4j.

KeyDB, a high-performance in-memory database acquired by Snap, offering over 1 million operations per second.

MeiliSearch, an open-source alternative to Elasticsearch, providing customizable full-text search capabilities.

MindsDB's integration of machine learning frameworks, allowing for predictive models to be embedded directly into the database.

Transcripts

if you want to be successful in Tech you

need to move fast and break things

that's exactly what I said to my boss

before he fired me for deleting a

production database I should fire

the relational database management

system is one of the most successful

ideas in the history of computing SQL is

1970s Tech that still forms the backbone

of the modern Big Data World Oracle

dominates Enterprise while MySQL is the

primary database for platforms like

YouTube and Facebook but SQL is by no

means perfect in 2023 there's a whole

new wave of futuristic Cutting Edge

avant-garde game changing databases

looking to move fast and break things

some of them build on top of existing

relational patterns While others bring

entirely new ideas to the table that

you've never seen before no pun intended

over the next few minutes we'll look at

15 crazy ass new databases and talk

about the problems they're trying to

solve first on the list we have Planet

scale which is not a new database but

rather a serverless platform for MySQL

like I mentioned before YouTube uses

MySQL however back in 2010 when it was

getting popular it was extremely

difficult to scale at least for an app

with millions of simultaneous

connections instead of switching to a

nosql document database they developed a

library called Vitas which makes some

trade-offs to scale MySQL horizontally

and this is the technology that planet

scales based on it provides a fully

managed database that can scale with

vtas and makes it available from a nice

dashboard with sdks to go along with it

and it also integrates nicely with tools

like Prisma it has a free tier and good

pricing on the page here that they're

likely losing money on by comparison a

basic database on Heroku is 50 bucks and

it's unreasonably expensive on AWS and

Google Cloud Planet scale is not the

only company with this serverless

database idea though yoga byte has its

own open source solution for postgres

its goal is to scale infinitely in the

cloud but attempt to support all the

native features in postgres which is not

easy to do in a cloud native environment

you can actually use a multi-cloud

strategy to host your database on

multiple clouds at the same time

eliminating the dreaded vendor lock-in

issue but there's another option for

scaling postgres called neon it's

currently in technical preview but looks

extremely promising it's serverless and

can scale down to zero and has a highly

polished admin dashboard to go along

with it and most importantly it's

written in Rust which is the only thing

that matters if you want to look cool in

front of your friends it also supports

branching which I think was inspired by

another database called dolt this

database would be like give MySQL and

git had a baby most importantly it

allows you to create branches from the

main database where you can experiment

with new data without breaking things it

makes it easy to analyze the diff

between changes and when everything

looks good the changes can be merged

back into the main branch and it's done

with the same conventions found in git

that virtually every developer is

familiar with dolt has a hosted option

but it is quite expensive they've raised

21 million dollars and because I'm

extremely entitled I demand they burn

more of this money to give us a free

tier I just paid eight dollars for a

dozen eggs and can't afford to put

another fifty dollars on my mom's credit

card now another sequel flavor I want to

talk about is cockroachdb it's been

around for quite a while and was

developed by ex Google employees and is

often referred to as new SQL it's

compatible with postgres but was built

from the ground up with go to scale

horizontally in the cloud it has a nice

free tier pretty similar to Planet scale

then becomes pay as you go pricing after

that it's yet another great option but

you might also want to consider

Foundation DB or titanium DB as well

which are also in this new SQL class but

in some cases you might prefer a

lightweight alternative like cloud flare

D1 this is a brand new product that's

still in beta and basically it's a SQL

Lite database that runs on the edge

cloudflare already has its own key value

store for workers as well as R2 for

object storage but D1 allows you to

create data on the edge that can be

queried one trade-off though is that

native transactions will not work

however they're taking a very cool

approach to overcome that limitation by

allowing developers to write stored

procedures with JavaScript which are

just Snippets of code that run directly

in your database which can be used to

create transactions or other reusable

queries this approach has a ton of

potential because historically store

procedures are a pain to work with but

if it's in JavaScript even I could do it

pretty cool but it's not production

ready yet if you want something that's

ready to go today check out Zeta yet

another serverless relational database

that's based on postgres and

elasticsearch under the hood but what

makes it special is that it treats your

data like a spreadsheet it feels kind of

like a developer friendly alternative to

airtable or notion and what's really

awesome is that it has full text search

built in without needing to duplicate

your data in some other service like

algolia it has sdks for typescript in

Python and has a schema editor that

makes it very easy to create and

visualize relationships between tables

and it can also do branching like we

looked at in adult DB for pricing it has

a solid free tier then it scales up in a

linear way based on usage but another

tool that takes things one step further

is eight base it's also a relational

database that provides a graphql API for

your data out of the box making it

similar to tools like hasura what makes

it unique though is that it also comes

with a low code tool for quickly

building front-end applications that

connect to your back end it's like a

backend as a service that can also build

your front end but that's enough talk

about relational databases now we enter

the post SQL ERA with Edge DB it's a

graph relational database that's powered

by postgres under the hood when it comes

to relationships humans don't naturally

think of columns and rows but rather

objects connected to each other like a

graph instead of defining tables like

you would in SQL you define types like

you would in most programming languages

these types link to other types which is

awesome because it eliminates the need

for joins and simplifies the way you

make queries overall all it almost feels

like an orm for postgres that allows you

to think about relationships in a far

more human friendly way currently it's

open source and can be self-hosted but

there's a wait list for their cloud

service now another relational graph

like database is surreal DB it's written

in Rust and I have a full tutorial on my

second Channel Beyond fireship it's

designed to support acid transactions

while scaling horizontally but what I

really like about it is how easy and

flexible it makes data modeling for the

most part it feels like SQL but also

uses arrows to connect nodes and edges

like you would in Cipher with a graph

database like neo4j it takes the best

elements of relational document and

graph databases and combines them into

one easy to use API currently it can

only be self-hosted but they do have a

cloud service coming soon the

multi-model idea is nothing new though

another database pushing the limits is

fauna it was created by X Twitter

Engineers who set out to build the

perfect database it's extremely easy to

use like a document database similar to

mongodb but unlike a document database

it supports features like native joins

which is the number one missing feature

in document databases and that means

fauna is more well suited for complex

relational data like social graphs to

interact with it it has its own custom

query language called fql but also comes

with a graphql API it is a closed Source

product so you can't self-host it but

its fully managed service has a nice

free tier another new database you've

likely never heard of is memograph it's

a drop in replacement for neo4j which is

the gold standard for graph databases

but memgraph is focused on building

real-time data pipelines like neo4j it

can be queried with Cipher but it's

written in C plus with a memory first

architecture that claims to achieve

better performance overall it looks like

it would be a great choice for building

something like a real-time analytics

platform next up we'll look at Key DB

which is a faster alternative to the

already extremely fast redis it was

recently acquired by snap a company that

needs to fetch data as fast as humanly

possible it can handle over 1 million

operations per second on a single node

which sounds like a lot to me but it's

not something you would use as your

primary database rather it's mostly used

as an in-memory cache to duplicate high

priority data from another database

making it load faster for the end user

it also has a library called mod.js

allowing you to extend the database with

your own custom commands and

functionality with JavaScript another

specialty database to look into is melee

search the full text search engine space

is dominated by tools like elasticsearch

and algolia a melee search is an open

source rust-based alternative that makes

it easy to implement an awesome search

feature into your app it's not a primary

database but instead a place where you

would duplicate data that needs to be

searchable in complex ways like you can

fine-tune different parameters to

control which attributes have a higher

ranking in the search in addition it has

a bunch of client libraries like instant

melee search that can quickly Implement

a search feature in a front-end web

application and finally that brings us

to mindsdb a database that integrates

popular machine learning Frameworks it

created a concept called AI tables that

embed predictive models directly into

the database at first glance it feels

like SQL except you can use it to write

statements that train machine learning

models when the model is done training

you can then use select to make

predictions from it so basically it

provides an into integrated way to

handle the ml Ops life cycle and it can

also integrate with your existing

machine learning models or those from

third parties like hugging face or open

AI in the not so distant future I

believe in betting AI directly into

databases will be a huge thing and

mindsdb gives you an idea of how that

could look and now you have 15 different

futuristic databases to choose from for

your next failed side project good luck

with that if I missed anything let me

know in the comments and if you want to

learn more about awesome serverless

databases like firestore and superbase

support my work as a pro member on

fireship iO thanks for watching and I

will see you in the next one