Hetu: A P2P Communication Layer With Causality Graphs

Summary

Jalen Lee, an assistant professor at NUS, introduces the HE2 protocol, a novel peer-to-peer communication layer designed for large-scale decentralized applications. HE2 offers a middle ground between traditional insecure P2P networks and secure but slow blockchains, providing trusted event ordering and verifiable logical clocks to ensure consistency and security without sacrificing scalability. The protocol leverages zero-knowledge proofs and incremental verifiable computation to create a community-driven platform with a flexible consistency model, suitable for various applications like social media, gaming, and decentralized infrastructure.

Takeaways

• 🎓 Jalen Lee, an assistant professor at NUS, is discussing the HAT2 protocol at V Labs.
• 🔗 HAT2 aims to generalize distributor consistency for large-scale decentralized applications.
• 📈 There's a spectrum of consistency levels and trust in peer-to-peer infrastructures, with limited options available.
• 🚀 Traditional TCP/IP and Li P2P offer fast communication but lack security and consistency.
• 🔒 Blockchains and permissioned BFT protocols provide strong security but are slow and don't scale well.
• 🌟 HAT2 offers a middle ground with low cost, high scalability, and various consistency levels.
• 🕒 The protocol leverages a novel verifiable logical clock for trusted event ordering.
• 🔄 HAT2 is a peer-to-peer communication layer that allows for building distributed applications with different consistency guarantees.
• 🔄 The verifiable logical clock uses zero-knowledge proofs and incrementally verifiable computation to ensure trust.
• 🛠️ HAT2 stack consists of a messaging layer, a shim layer, and an application layer for community-driven development.
• 📅 The HAT2 team plans to launch a testnet in mid-year and mainnet by the end of the year or early next year.

Q & A

What is the main focus of the HAT2 protocol?

The HAT2 protocol focuses on generalizing the notion of distributor consistency for building large-scale decentralized applications, providing a middle ground between traditional peer-to-peer networks and blockchain technology in terms of consistency and security.

What are the limitations of traditional TCP/IP stack or Li P2P in terms of security and consistency?

Traditional TCP/IP stack or Li P2P provides fast networking and communication between peers but lacks significant security and consistency, offering only basic security features like end-to-end encryption and message integrity.

How does the HAT2 protocol address the scalability issues of blockchain technology?

HAT2 protocol leverages a novel verifiable logical clock to provide trusted event ordering at a low cost and with high scalability, allowing for the development of distributed or decentralized applications without the scalability bottlenecks of blockchains.

What is the significance of the verifiable logical clock in the HAT2 protocol?

The verifiable logical clock is a key component of the HAT2 protocol that allows for the establishment of causal ordering of events in a distributed system without relying on physical clocks. It ensures that the logical causality can be verified by any third party, preventing malicious users from creating false causal dependencies.

How does the HAT2 stack structure its layers?

The HAT2 stack is structured into three layers: the messaging layer for creating and verifying proofs for logical clocks, the shim layer for building middleware like relay networks and sequencers, and the application layer where developers can propose and build various applications using the logical clocks.

What is the HAT Improvement Proposals (HIP) system?

The HAT Improvement Proposals (HIP) system is a community-driven approach that allows anyone, not just the core development team, to propose improvements or new components for the HAT2 protocol. It is similar to the EIP system used in Ethereum.

What are some potential applications of the HAT2 protocol?

Potential applications of the HAT2 protocol include decentralized mutual exclusion, causally and eventually consistent datastores, virtual machines, and shared sequencing layers. It can also be used for social applications, decentralized games, and other interactive applications requiring a balance between consistency and scalability.

What is the timeline for the HAT2 protocol development?

The current proof of concept (POC) is being developed, with plans to launch a testnet in the middle of the year, followed by a mainnet release by the end of the year or early next year.

How can developers contribute to the HAT2 protocol?

Developers can contribute by proposing improvements or new components through the HIP system. Once approved, their contributions can be merged into the HAT2 stack. They can also develop applications using the provided APIs and share their ideas with the community.

What are the challenges of maintaining a common physical clock in a distributed system?

In a distributed system, maintaining a common physical clock is challenging due to the absence of a shared reference frame, as different nodes may have different relative motions (time dilation) and clock drifts, making it difficult to synchronize clocks accurately.

How does the HAT2 protocol ensure consistency without a common physical clock?

The HAT2 protocol uses a logical clock based on causal ordering of events, which does not rely on physical time but instead tracks the causal dependencies between events. This allows for a consistent ordering of events without the need for a common physical clock.

Outlines

00:00

🎤 Introduction to Hat Protocol

Jalen Lee, an assistant professor at the National University of Singapore, introduces the Hat2 protocol. He discusses the challenges of building large-scale decentralized applications with existing infrastructures, which either offer high speed with minimal security or strong security with limited scalability. The Hat2 protocol aims to provide a middle ground, offering more consistency and security options for developers.

05:01

🔄 Event Ordering in Distributed Systems

The importance of event ordering in distributed systems is highlighted, with traditional methods relying on physical clocks. In decentralized systems, a common physical clock is absent, leading to the concept of causal ordering. Lamport's work on logical clocks and event ordering in distributed systems is referenced, emphasizing the need for a trusted way to order events without a physical clock.

10:02

🔒 Verifiable Logical Clock

A new form of verifiable logical clock is introduced, which allows for event ordering in a peer-to-peer environment with potential adversaries. This clock uses zero-knowledge proofs and incrementally verifiable computation to ensure that the logical clauses can be verified by any third party, preventing false logical clocks and maintaining causal dependencies.

15:03

🌐 HE2 Stack and Community Approach

The HE2 stack is described as a peer-to-peer communication layer that provides trusted event ordering. It consists of a messaging layer, a relay network, and an application layer with a simple API. The HE2 protocol is community-driven, allowing developers to propose improvements. The current proof of concept includes a verifiable logical clock, a peer-to-peer communication library, and demo applications like Z Social. The timeline for the HE2 stack includes a testnet and mainnet release in the coming year.

Keywords

💡Distributed Consistency

Distributed Consistency refers to the ability of a distributed system to maintain a consistent state across multiple nodes. In the context of the video, it is a core challenge for building large-scale decentralized applications, where the system must ensure that all participants agree on the order and state of events, even when they may not have a common physical clock or may be subject to adversarial conditions.

💡Decentralized Applications

Decentralized Applications (DApps) are software applications that operate on a decentralized network, often using blockchain technology to ensure trust and security without a central authority. The video emphasizes the need for such applications to have a balance between security, consistency, and scalability, which the hat2 protocol seeks to address.

💡Peer-to-Peer (P2P) Infrastructure

A Peer-to-Peer (P2P) Infrastructure is a network where each participant (peer) has the same capabilities to share and receive resources. It contrasts with client-server models where resources are centralized. The video script discusses the limitations of P2P infrastructures in terms of security and consistency, which the hat2 protocol aims to improve.

💡Verifiable Logical Clock

A Verifiable Logical Clock is a system that tracks the order of events in a distributed system without relying on physical time. It uses logical or causal relationships between events to establish a consistent order. In the video, this concept is crucial for the hat2 protocol, as it allows for trust and consistency in a decentralized environment, even when nodes may be adversarial.

💡Zero-Knowledge Proof

A Zero-Knowledge Proof is a cryptographic method that allows one party to prove to another that a statement is true without revealing any information about the statement itself. This concept is integral to the hat2 protocol's verifiable logical clock, as it enables the creation of a trusted event ordering without compromising privacy.

💡Incrementally Verifiable Computation (IVC)

Incrementally Verifiable Computation (IVC) is a technique that allows multiple parties to collaboratively compute a result while verifying each step of the computation. This ensures that the final result is correct and trustworthy. In the context of the hat2 protocol, IVC is used to build a trusted logical clock that can be verified by any participant in the network.

💡Causal Consistency

Causal Consistency is a consistency model in distributed systems that ensures the order of events respects the causal relationships between them. This means that if an event A causes an event B, then A must occur before B in the system's view. The video emphasizes the importance of causal consistency for applications that require a certain level of trust and consistency without full global consensus.

💡Hat2 Protocol

The Hat2 Protocol is a novel peer-to-peer communication layer designed to provide trusted event ordering and various consistency levels for building distributed or decentralized applications. It aims to offer a middle ground between traditional P2P networks and blockchains, addressing the limitations of both in terms of security, consistency, and scalability.

💡Community-Driven Approach

A Community-Driven Approach refers to the development process where a community of contributors, not just a core development team, can propose and contribute to the growth and improvement of a project. The video script highlights the Hat2 protocol's adoption of this approach, allowing for a more collaborative and open development process.

💡Proof of Concept (POC)

A Proof of Concept (POC) is a demonstration that a particular method or idea is feasible and can be successfully implemented. In the context of the video, the Hat2 protocol has a POC that showcases its capabilities and potential for future development.

Highlights

Introduction to the hat2 protocol for building large scale decentralized applications.

Challenges in achieving consistency and security in peer-to-peer infrastructures.

The spectrum of consistency levels and trust in infrastructures, from traditional TCP/IP to blockchains.

The need for a middle ground between fast but insecure networks and slow but secure blockchains.

HE2 protocol as a peer-to-peer communication layer offering trusted event ordering.

Utilization of a novel verifiable logical clock for event ordering in a decentralized environment.

The concept of causal ordering of events in distributed systems without a common physical clock.

The development of a new logical clock that doesn't depend on physical clocks but on causal dependencies.

The application of zero-knowledge proofs for incrementally verifiable computation in building a trusted logical clock.

Examples of applications that can benefit from the verifiable logical clock, such as decentralized mutual exclusion and causally consistent datastores.

The structure of the HE2 stack, including the messaging layer, shim layer, and application layer.

The community-driven approach of the hat2 protocol, allowing developers to propose improvements.

Proof of concept development and the roadmap for the HE2 stack, including a testnet and mainnet release.

Invitation for community contribution and feedback, with team members present at the event.

Transcripts

[Music]

all right um so yeah today I'm very

excited to uh share with you guys uh our

hat to protocol um so forgive me I'm uh

I just recently got a pretty bad code so

my voice is a bit like a bit shaky uh

but hope hope everybody can hear me is

it good oh good okay cool all right so

uh first a quick introduction uh so my

name is Jalen Lee uh so I'm currently an

assistant professor at the National

University of Singapore and also I'm

representing uh at the V Labs uh to uh

to tell you more about our uh very

recent hat2 protocol uh where we try to

generalize uh this generalized notion of

distributor consistency uh for building

uh large scale decentralized

applications all right um so let me

first start with I guess all of you

heers are

uh our developers uh in the web stre

space so uh you you guys use or have

been uh developing infrastructure or use

large scale peer-to-peer

infrastructures um but so far there are

very limited uh number of options in

terms of what kind of consistency you

get uh or kind of security you get from

this infrastructure right so I here I

gave you sort of like a graphical

illustration the kind of spectrum of

consistency levels or the kind of trust

you get from this

infrastructures so on the very uh left

far end of the spectrum you get things

like traditional TCP IP stack or Li P2P

or things like that where you can build

very fast um very fast networking right

uh communication between different peers

in a uh in a peer-to-peer Network right

but the problem is that you don't really

get a lot of security or consistency

from these infrastructures right so you

kind of get uh the bare minimum kind of

security if you do end to endend

encryption right so that gives you

things like authent authentication

Integrity of your messages but you don't

get much more than that right uh so if

you build distributed applications or

decentralized applications would require

a lot more security or consistency uh

instead what you do is you choose the

far right end of the spectrum where you

use probably a blockchain use a uh

permission bft protocol or you use any

other form of infrastructure like Layer

Two rops reaking and other forms of

instructure that gives you very very

strong security right so you have a

totally ordered uh uh set of blocks

right so everybody agrees on the order

so there's Global consensus right uh so

security wise is really good uh it's a

very strong consistency uh but the

problem is it's very slow it doesn't

really scale right so for many kind of

applications it's not probably not ideal

to use this infrastructure as well right

but we can see that this is a spectrum

right so you don't only have two end

points there's a lot of ground in the

middle right so all these middle uh for

Security Options and consistency options

they're currently not a lot of uh for

you infrastructure wise for to use right

um but actually we see that there are a

lot of applications which can actually

benefit from a more uh slightly more

relaxed level consistencies or trust

models um to to scale them right so here

I just gave you a sense of uh example

application but here definitely there

are not a uh exhaustive list right so

here things like social applications

right things like communication between

different AI agents right uh like Games

social games um uh and more recent like

uh uh decentralized physical

infrastructure and so on so forth right

so all these applications right they are

so they're they're proper one of the

main properties that very interactive

right so there are a lot of interactions

between different parties in the network

a lot of communications going on right

so putting them on chain is probably not

the best option because of the

scalability bottleneck so even though

you have more scalability options these

days uh but eventually because the cost

and then there's still limited

throughput and scalability is still not

ideal for you to to to build these

applications on top on the other hand

these application do require some level

of security and trust and also

consistency right so for example if

you're building social applications you

want to have some kind of consistency so

that your followers right your tweets

right so there are there there there the

follow your followers right so these are

your your your uh your metadata changes

there there are some kind of consistency

guarantees right so so there's so at

this moment you either choose right the

uh the very uh very uh so infrastructure

that they don't give you anything or you

you choose like blockchains uh so which

are fairly Limited in terms of

scalability right uh so in this talk so

we're going to introduce HE2 uh which

does provide this kind of Middle Ground

where there are offers a lot more kind

of consistency and Security Options for

you to build distributed or

decentralized applications right so HE2

is a peer-to-peer communication layer

right so but uh so compared to existing

peer-to-peer infrastructures or

communication infrastructure uh it

additionally provides this what we call

trusted event ordering uh naturally from

this communication layer right uh and

then underlying it we are leveraging a

novel um form of verifiable logical

clock uh which we I'm going to elaborate

uh in a few minutes um so the so the so

the so the nice thing about HE2 is that

it is very low cost and then very highly

scalable um but also it gives provides

the opportunity for distributed or

decentralized application to build

various consistency levels on top of it

uh and then so it has really really wide

kind of applic applicability right so

all the applications I just show you uh

will will benefit from our communication

stack so let me just elaborate by

starting from a little bit background

right so uh so when you build any kind

of applications distributed systems

right so the order of events is a really

important aspect right so no matter what

you do you care about some kind of

ordering right either is like between

different blogs or not between a uh a

social social media post and then some

other activities right so these are all

events and you care about the ordering

so how do you order events in a system

right so the most traditional way so for

example if you come to uh this e Denver

event where you look at all the events

uh in this in this uh in on the schedule

so you clearly Mark right what is what

time is each event happening right so

that's the most traditional way of

ordering events right so my talk starts

at 1:45 and then the previous stock

start at like 1 uh 1 p.m. or 1:30 p.m.

or something right so by looking at the

different time span time stamp you can

know exactly which event happened first

so but the problem is that in a truly

distributed or decentralized system this

kind of physical clock or commonly

agreed physical clock is not present so

there are multiple reasons why you don't

have that in a decentralized

system so first from a theoretical

perspective uh if you guys uh have heard

or learned about relativity right so in

relativity right so different objects uh

which have different relative motions

they actually don't have the same common

clock right uh so this phen phenomena is

called time dilation so for example uh

here I show uh on the left is the

international uh space uh station right

so and then so because it's moving with

a relative motion with for example this

is the location of e dver right because

if if there is relative motion so the

time actually disagree with each other

right so they don't have the common

time so the other problem is that even

if you have if you don't have relative

motion among different objects because

your clocks devices they are not perfect

right so they cannot track your physical

time um very accurately right um so for

example depending on what kind of clock

technology you use if you use U small

atomic clocks right versus very uh so

sorry very sophisticated atomic clocks

versus a simple clock you can find your

computer like a crystal oscillator so

they have very different clock drifts

right so it's very hard for you have to

have perfectly synchronized clocks among

different

parties right so if we can't have like a

a common agree upon physical clock what

can we do to have uh to to order events

in a distributed system so here we look

at uh look back in the literature in

distributed systems so back in 1978 uh

so a famous distributed system

researcher uh his name is Lassie Lamport

uh who later got theer award for some of

this work including this so uh he

published a seminal work called time

clocks and The Ordering of events in

distributed system where he exactly talk

about this problem so if you don't have

physical clock in the system how can you

order events so he argued that the only

trusted way of ordering events in a

distributed system is to look at The

Logical ordering of different events or

another way of saying it is called the

Cal causal ordering of events in a

system so what is causal event ordering

in a system so here I show if you have

multiple processes in a distributed

system right so they talk to each other

through messaging right so you don't

have a clock so you don't know which

event happened first but you know for

sure that if I send the message from

process P to process Q the sending of

the event must happen before the

receiving of the message now I establish

a causal dependency between these events

and then you can use this if I I receive

a message and later send another event

uh and sorry I send another message I

know the sending of the next message

causally depending on I received the

previous message right because the

sending May cly depend on actually

receive the previous message right and

then you can do a transitive closure of

these events then you get this what they

called a happened before relationship

which captured the causal orological

dependency between different events so

that you don't require any physical

clock and then that is the true ordering

of the the system that nobody can

disagree with right so that's a pretty

cool and a pretty strong kind of uh uh

event ordering property and then uh

let's lample argue that now we can

develop a new kind of clock that doesn't

depend on physical clock but instead on

a logical clock this logical clock

essentially track this causal uh causal

dependency between different events by

looking at this causal clock sorry

logical Clause you can infer what's the

causal dependency between different

events right so so this paper and the

notion of logical clock on causal

dependency have a lot of implication a

huge implication on the development of a

lot of distributed and decentralized uh

protocols to this state um so we are uh

taking this protocol and this notion a

step forward so we are looking at how do

we build logical clock in a

decentralized environment right so a

peer-to-peer environment one of the

biggest challenges in this peer-to-peer

envir environment is the presence of

potentially malicious or adversaries

users right so pantin knows or another

way of saying it right so because

traditional logical clock depends on all

the protoc all the all the participant

in the system follow exactly the

protocol of updating The Logical clock

right so if there are any pantin node in

the system it will break the break the

entire notion of logical clock and the

caal dependency guarantees right so what

we do is we have done some uh recent re

research and and then we found or we

developed a new way a new logical clock

called verifiable logical clock uh that

ensures that the The Logical Clause pass

in the system can be verified by any

third party right so that means nobody

can can uh create a false logical clock

that breaks Lo uh causal dependencies

right and then so I mean I'm not going

to have the time to give you all the

technical background uh but a hell of a

gist is that we leverage this new in

noal application of zero knowledge proof

so this is very recent uh development in

the space uh so this new technique is

called incrementally verifiable

computation or IVC that allows us to

have multiple parties in the system by

passing these messages build a trusted

logical clock with a verifiable

proofs right and by applying this uh

this novel kind of verifiable logical

clock we're able to build a lot of

interesting applications and

infrastructure on top of it here I just

give you a some quick uh examples that

we recently developed uh but you can

certainly think of using this verifiable

logical clock for a lot of more

interesting applications right so for

example we can use uh this verifiable

logical clock to build decentralized

Mutual exclusion or the other way of

saying is like m a a lock right uh a

synchronization primitive between

different participant in the system

without centralized trust or centralized

server for log Services right and you

can build as I I hinted in the beginning

of the talk right so there are a lot of

application that I want something uh

consistency levels not that are not like

strictly consistent right not

linearizable in a more technical sense

right so here we are able to use this uh

verifiable logical clock to build

causally and eventually consistent data

stores and then uh uh things like

virtual machines right so your virtual

machine State doesn't need to be always

synchronized across each other but we

guarantee either eventually synchronized

or if there's a c dependency between the

different updates we make sure everybody

observed them in a causally consistent

order right and then we can use this

logical verifiable logical clock to

build a shared sequencer for example

right here we're able to use the partial

ordering property of this verifi logical

clock to easily scale a shared

sequencing

layer right uh and then so so far we

have built a so from this verif a

logical clock we build a HE2 stack uh so

this HE2 stack as I mentioned before is

a peer-to-peer communication layer right

um so sorry that the the the picture

might be a little bit blurry and small

uh So currently we we can structure as

three three kind of layers so the first

layer is underlying the the lowest layer

we have this messaging layer where um so

we can the messaging layer creates the

uh the proof and then generate generate

proofs and then verify uh the for the

for this logical clocks and verify this

logical clock while generating messaging

and then this layer also uh uh build

build a traditional p-to-p communic

layer things like cadamia uh DHT right

and then on top of it we have a a middle

what we call the shim layer where we

build a sort of like a a relay Network

where there's a large scale uh

peer-to-peer network with a lot of

participants messaging each other uh

with uh the underlying shared sorry

underlying a verifiable logical clock in

this each messages and on top of it

expose a simple to use API to the

application layer where the application

can use uh the The Logical clocks from

the underlying layers to build their uh

desired consistent levels and their

applications so that's the current hat2

protocol uh hat to stack uh and then we

are a very Community Driven approach

right so so we uh uh we develop a fairly

interesting and pretty novel uh protocol

called the hat to Improvement uh

proposals this is like the EIP system

right where everybody not just the our

core development team can propose um

different Improvement on top of K to

protocol or a components of the hatter

protocol right so here we gave you some

we roughly coarsely developed into three

different layers so the first layer is

called hiip a layer so this is the like

the fundamental layers of the the HE2

protocol so here so we build the

fundamental technology behind HE2 things

like logical clocks uh the peer-to-peer

Network and then uh the incrementally

verifiable verifiable computation right

and then on top of it there is the hip B

layer where we there's the middle uh

middleware that people can build things

like the relay Network I mentioned about

and also things like sequencers and then

random number generators and then there

are many developers who care more about

the upper level applications and then

they can propose uh things like

Messengers for social applications uh

decentralized games or things like

social other kind of social applications

on top of it right so we open up uh this

hip system to all the developers in the

system uh in the in the community right

everybody can propose right and then

once it's approved uh they can be merged

into uh the Hato

stack right so currently we have a proof

of concept right so if you're interested

feel free to check uh check out our uh

GitHub page right so currently we have a

few of these components currently

already built things like The verifiable

Logical clock so the underlying

peer-to-peer communication Library uh

and then we also developed some a few

applications on top of it already as a

demo right things like uh a social

application called Z social and then a

random number

generation right so um so this is our

prep proposed timeline for our uh HE2

stack and HE2 protocol uh so I don't

know if you guys can read it but we are

currently roughly here so we're uh we

develop our current propal concept POC

and then our goal is to have the test

net some sometime uh in the middle of

this year right so which is here uh and

then later this year and by the end of

this year or early next year where our

plan is to row out the main net of hat2

uh so we welcome everybody to uh

contribute uh to talk to us give us

feedback

um so uh so many of our team members are

here at East Denver so if you are

interest in our protocol or our stack

feel free to talk us uh so uh so aasha

is here in this I think somewhere here

uh that's me and then I think Sean is

also here and then we have many other

people here from the uh the hat2 team uh

and then uh also uh follow us on Twitter

uh and then we welcome all your feedback

and then yeah I think that's it and

thanks everybody

bu