eBPF’s Abilities and Limitations: The Truth - Liz Rice & John Fastabend, Isovalent

Summary

In this engaging session, Liz Rice and John Fastabend introduce and explore the capabilities of eBPF, a technology allowing custom programs to run within the Linux kernel. They debunk myths about eBPF's limitations, demonstrating its power by implementing the Turing-complete 'Game of Life' directly in eBPF. The discussion covers the evolution of eBPF, its current state, and its potential for complex tasks, emphasizing the vibrant community driving its continuous improvement.

Takeaways

• 😀 Liz Rice and John Fastabend are presenting on BPF (Berkeley Packet Filter), emphasizing its power and potential for custom kernel behavior modification.
• 🔧 BPF allows for running custom programs in the kernel, which can change the kernel's behavior by attaching to events, making it a versatile tool for system operations.
• 🖥️ The kernel is central to operating systems, involved in hardware interaction, file access, network communication, and process management, making BPF's impact extensive.
• 🛠️ There are numerous infrastructure tools built using eBPF, such as Cilium and Tetragon, which focus on networking, security, and observability.
• 🚫 Common misconceptions about BPF's limitations, such as its inability to handle complex tasks like layer seven packet parsing, are being challenged in the presentation.
• 🔄 The concept of 'Turing completeness' is discussed, highlighting that while BPF is not Turing complete, it is still capable of processing complex tasks within certain bounds.
• 🔒 The BPF verifier is crucial for ensuring safety when BPF programs are loaded into the kernel, checking memory access, control flow, and preventing indefinite loops.
• 🔄 BPF has evolved to include features like loops and timers, which enable more complex and long-running programs to be safely executed within the kernel.
• 💡 The demonstration of running Conway's Game of Life within BPF showcases the capability of BPF to handle complex, ongoing computations.
• 📈 The BPF community and ecosystem are continuously growing, with ongoing work to improve the verifier, compiler, and overall usability of BPF.
• 📚 Resources for learning more about BPF, including books and labs, are available, and the presenters encourage engagement with the community for advancing BPF capabilities.

Q & A

What is eBPF and why is it significant in the context of the kernel?

eBPF, or Extended Berkeley Packet Filter, allows custom programs to run in the kernel, enabling changes to the kernel's behavior by attaching these programs to events. It's significant because the kernel is involved in all hardware-related operations of an operating system, such as file access, network communication, and memory allocation, as well as managing system processes and permissions.

What is the role of the kernel verifier in eBPF programs?

The kernel verifier checks eBPF programs to ensure they are safe to run. It verifies that the program can only read and write to allowed memory, has valid control flow, does not get stuck on the CPU, and properly manages locks and references.

Why was the concept of Turing completeness brought up in the discussion about eBPF?

Turing completeness was discussed to address misconceptions about the capabilities of eBPF. It was used to illustrate that eBPF can, in theory, process any task that a Turing machine can, given it has the ability to process an arbitrary amount of data and time and store states.

What is the Game of Life and how does it relate to eBPF?

The Game of Life is a cellular automaton that evolves based on a set of rules determined by the state of neighboring cells. It was used in the script to demonstrate that eBPF is capable of running complex tasks, as it was successfully implemented within an eBPF program.

How has the eBPF verifier evolved to accommodate more complex programs?

The verifier has evolved to support loops, allowing programs to repeat operations without the risk of running indefinitely. It also supports callbacks and iterations, enabling eBPF programs to perform tasks over time without locking up the CPU.

What are the practical limitations of eBPF programs in terms of instructions and loops?

While eBPF programs were initially limited to 4,000 instructions, this limit has been increased to 1 million instructions in newer kernel versions. Loops are allowed as long as they can be verified to terminate, ensuring the program does not run indefinitely.

How can eBPF programs handle memory allocation?

eBPF programs can handle memory allocation through the use of array maps, which are memory blocks allocated by the user space and given to the eBPF program. The size of these maps is limited by the system's available memory and the user space's allocation limits.

What are some of the practical benefits of not being fully Turing complete in certain eBPF applications?

Not being fully Turing complete can be beneficial in scenarios where bounded execution is desired, such as in parsers where an upper limit on execution time can prevent infinite loops, or in systems monitoring where it's important to avoid long-running processes that could impact system performance.

What is the relationship between eBPF and infrastructure tools like Cilium and Tetragon?

eBPF is used in infrastructure tools like Cilium and Tetragon for networking, security, and observability tasks. These tools leverage the capabilities of eBPF to perform complex operations within the kernel, improving efficiency and performance.

What is the future outlook for eBPF in terms of development and community growth?

The future outlook for eBPF is positive, with continuous improvements and evolution of the technology. The community is growing, and there is a vision of pushing more processing into the kernel, which could lead to even more powerful and efficient applications in areas like networking and system monitoring.

Outlines

00:00

📘 Introduction to BPF and Session Overview

The session begins with an introduction by Liz Rice and John Fastabend, who express excitement about discussing BPF (Berkeley Packet Filter). John, an expert in eBPF with nearly a decade of experience, clarifies that eBPF allows custom programs to run within the kernel, influencing its behavior by attaching to events. The kernel's role in operating systems is highlighted, emphasizing its involvement in hardware interactions, file access, network communication, and process management. The speakers aim to debunk misconceptions about eBPF's capabilities, particularly its alleged limitations in handling complex tasks like layer seven packet inspection and its supposed incompleteness compared to general-purpose programming languages.

05:01

🔍 Exploring eBPF's Potential and Limitations

The speakers delve into the concept of Turing completeness, explaining it as the ability to process any arbitrarily complex task, data, or time without bounds. They contrast this with eBPF's design, which includes intentional limitations to ensure safety and performance, such as bounded execution times. Examples of both useful and less useful Turing complete systems are given, including the Game of Life, which is Turing complete but not necessarily practical for certain applications. The benefits of bounded properties in eBPF are discussed, such as preventing infinite loops in parsers or kernel functions, which can be crucial for maintaining system stability.

10:01

🛠 The Role of the eBPF Verifier in Safeguarding Kernel Programs

John Fastabend explains the role of the eBPF verifier, a component that ensures the safety of programs running in the kernel. The verifier checks for memory access permissions, control flow integrity, and ensures that programs do not get stuck on the CPU indefinitely. It also verifies that locks and references are properly accounted for. The discussion highlights the evolution of eBPF, including the addition of features that allow programs to sleep, handle callbacks, and iterate, which have expanded the capabilities of eBPF beyond the original model.

15:02

🎲 Demonstrating Game of Life in eBPF

The speakers present a live demonstration of Conway's Game of Life implemented in eBPF, showing that complex, Turing complete tasks can be executed within the kernel. The demonstration involves a sensor added to Tetragon, an eBPF project, which generates and evolves a pattern of 'cells' according to the rules of the Game of Life. The core logic of the game is briefly explained, including the copying of cell states, iteration of rules, and updates sent to user space for visualization.

20:03

🔧 Evolution of eBPF and Overcoming Instruction Limits

The discussion addresses the evolution of eBPF, particularly the increase in the number of allowable instructions from 4,000 to 1 million, to accommodate more complex programs. The importance of loops in eBPF is highlighted, as they allow for more sophisticated program logic while still ensuring termination. The speakers also touch on the challenges of writing eBPF programs that are verified by the kernel, emphasizing the need for synchronization between human programmers, compilers, and the verifier.

25:03

🌐 Future Prospects and Encouraging Community Engagement

The session concludes with a look towards the future of eBPF, emphasizing its continuous improvement and expansion. The speakers encourage the audience to engage with the eBPF community, suggesting that with enough interest and collaboration, even more complex and diverse applications can be developed using eBPF. They highlight the potential for eBPF to handle more networking-related processing in the kernel and mention resources for learning more about eBPF, including books, labs, and community projects.

Keywords

💡eBPF

eBPF, or Extended Berkeley Packet Filter, is a revolutionary technology that allows for the running of custom programs within the Linux kernel. It is central to the video's theme as it enables the modification of kernel behavior through the attachment of eBPF programs to various kernel events. The script discusses the evolution and capabilities of eBPF, demonstrating its power through the implementation of complex tasks like the Game of Life directly in the kernel.

💡Kernel

The kernel is the core part of an operating system that has control over everything the system does, including hardware access, file operations, and network communication. In the context of the video, the kernel's significance is highlighted by its role in eBPF programming, where eBPF programs can be attached to kernel events to alter system behavior.

💡Turing Completeness

Turing completeness refers to the ability of a system to perform any arbitrary computation given unlimited time and resources. The video discusses the concept in relation to eBPF, addressing misconceptions about the limitations of eBPF and demonstrating that it can handle complex tasks indicative of Turing completeness, such as running the Game of Life.

💡Verifier

The verifier in the context of eBPF is a component that checks the safety of eBPF programs before they are loaded into the kernel. It ensures that the programs adhere to certain constraints, such as not reading unauthorized memory or causing the kernel to crash. The script explains how the verifier has evolved to support more complex eBPF programs, including those with loops.

💡Game of Life

The Game of Life is a cellular automaton devised by the mathematician John Conway. It is used in the video as a practical demonstration of the capabilities of eBPF, showing that it can execute a Turing complete program. The script describes how the Game of Life was implemented in eBPF, emphasizing the flexibility and power of eBPF for complex tasks.

💡Instruction Limit

The instruction limit in eBPF refers to the maximum number of instructions an eBPF program can have. Initially set to 4,000, this limit has been increased to accommodate more complex programs. The script discusses how the evolution of eBPF has led to an increase in this limit, allowing for more sophisticated programs to be written.

💡Looping

Looping is a programming construct that allows for the repetition of a block of code. The script explains how early versions of eBPF did not support loops due to verifier limitations but have since evolved to include this feature. This advancement is crucial for handling more complex tasks within eBPF programs.

💡Cilium

Cilium is an open-source project that utilizes eBPF for providing networking, security, and observability for microservices. It is mentioned in the script as an example of infrastructure tools built using eBPF, showcasing the practical applications of eBPF in the industry.

💡Tetragon

Tetragon is a project that is part of the script's discussion, used to demonstrate the capabilities of eBPF. It is shown to be capable of adding new sensors, such as the Game of Life, and is used to illustrate the flexibility and extensibility of eBPF in real-world applications.

💡Memory Management

Memory management in eBPF involves how programs handle memory allocation and access within the kernel. The script touches on this topic, explaining the use of array maps for memory blocks and the importance of ensuring that eBPF programs do not exceed the available system memory.

Highlights

Introduction to BPF (Berkeley Packet Filter) and its evolution to eBPF (extended BPF), allowing custom programs to run in the kernel and modify its behavior.

John's 10-year experience with eBPF and his role as an engineer in night surveillance, emphasizing his expertise in the field.

Explanation of the kernel's role in operating systems, particularly its involvement in hardware interactions, process management, and permissions.

Discussion on the capabilities of eBPF to change kernel behavior and its use in infrastructure tools for networking, security, and observability.

Misleading statements about the limitations of eBPF, such as its inability to implement complex layer seven packet inspection, debunked by the speakers.

Introduction of the concept of 'Turing completeness' and its relevance to eBPF's ability to process complex tasks and data.

Examples of Turing complete systems, including general-purpose programming languages and the Game of Life, to illustrate the concept.

Benefits of not being fully Turing complete, such as bounded processing for safety and performance in certain applications.

Live demonstration of running the Game of Life within eBPF, showcasing the capabilities of eBPF beyond traditional expectations.

Details on eBPF verifier's role in ensuring safety, memory access control, and preventing indefinite loops in eBPF programs.

Evolution of eBPF to support loops and iterations, overcoming early limitations and allowing more complex programs to run.

Comparison of eBPF instruction limits with real-world programs like Doom, Envoy, and Clang, highlighting the practicality of eBPF for large-scale applications.

Explanation of subprograms in eBPF and how they relate to function calls, enabling large programs to be executed within the kernel.

Discussion on the future improvements in eBPF, including better support for loops and increased ease of programming.

Practical applications of eBPF in cloud providers and distributions, indicating the growing adoption and integration of eBPF in the industry.

Resources for learning more about eBPF, including books, labs, and community projects, encouraging further exploration and contribution.

Final thoughts on the potential of eBPF, emphasizing its current capabilities and the ongoing efforts to expand its utility and accessibility.

Transcripts

good morning thank you so much for

coming to our session this morning just

about still morning right uh I hope

everybody's able to find a seat if you

have got a seat next to you maybe

shuffle along and and let people in uh

yes so my name's Liz rice uh myself and

John here do you want to say hi yeah hi

my name is John fman I'm engineer night

surveillance and excited to talk about

BPF today I've been you know working on

it for almost 10 years now so hopefully

you guys get as excited as um as I am

yeah

so I mean as John says he's he works in

the Kel right he is a proper expert in

ebpf let's just quickly recap to make

sure everybody understands what we're

talking about with ebpf it allows us to

run custom programs in the

kernel so we can change the way the

kernel behaves by loading ebpf programs

there and attaching them to events why

is the kernel interesting well the

kernel is the part of the operating

system that is involved whenever we're

doing anything with Hardware so if

you're accessing a file or sending

receiving over a network or uh even

allocating memory the Kernel's going to

be involved the Kernel's also um

managing the processes running on a

system so it's looking after things like

permissions and privileges so the kernel

is involved in pretty much everything

interesting with ef we can change the

way that it

behaves so already there are lots of

really great infrastructure tools built

using ebpf I mean we're both very

involved in psyllium and uh cium sub

project tetragon doing things like

networking security observability there

are also loads of other great projects

out there and great products out there

that are doing things in these kind of

infrastructure tooling spaces

but sometimes you'll hear statements

that give the impression that there are

limits to what you can do with ebpf so

for example we've heard people saying

yeah you can't really Implement like

layer seven paing that's just too

complex for ebpf or another thing that

you'll quite often hear people saying is

yeah but there are limits to what you

can do with ebpf because it's not cheer

and complete right so do we think those

statements are true well a hint here is

these things were not said by people who

are working on

ebpf we're going to explore these kind

of statements

today so I've said the term cheering

completeness this is not going to be a

computer science class it's not going to

be like super pedantic about

terminology but really what true

incompleteness about is about is can we

process some on arbitrarily complex task

can we process an arbitrary amount of

data can we continue processing for an

unbounded amount of time do we have the

ability to store States so that we can

continue sort of process state so this

is broadly speaking what true

incompleteness is about

right so I I think it's interesting um

just to talk a little bit about some

concrete examples right I mean we could

talk about the like mathematical

formulation of turn complete but that's

perhaps not interesting to everyone here

so like typically when we think about

turning completeness people really

talking about like your general purpose

languages C C++ Java python go all the

things that we um programmers love um

but I I also want to point out that

there's a lot of like really also

interesting but maybe less interesting

for programmers you know nobody really

wants to write um your uh application in

the one instruction set computer so

that's a totally valid turing machine it

is Turing complete but you know try to

write a proxy with just the move

instruction and it might not be quite as

fun as you want um another example is

The Game of Life that will go back to

but there's just loads of these things

and you know the the point is try not to

equate I think turn completeness um with

sort of usefulness as a programming

language to um you know developers right

um I also want to note that there's some

like actual benefits of not being fully

tur and complete sometimes um you might

be interested for example in having a a

parser um that is bounded and why would

you want that you might want that

because you don't want to receive a

packet and then have your parser Loop

forever really would like the thing to

stop at some point um you know it's the

same way like in BPF space where we do

like CIS call and tetron we'll watch CIS

calls or watch kernel functions right we

really like to have an upper bound on

the time that we're going to kind of

impact the application or the operating

system and when you do that you want to

say like it's really nice to say you

know this will not run longer than a

millisecond or you know 500 NCS or

something like this right and that

really is not touring Complete because

we're putting a bound on that

environment but um there you know

there's really concrete advantages to

that as well so just want to you know

kind of lay that out there and and kind

of give you some examples of of turning

complete things that are both useful and

not useful and an example of maybe why

you you know would want some bounded

properties as well

and true and completeness talks about an

infinite tape right and there is no

infinite tape in the world so in reality

we're always dealing with just large

rather than infinite yep

exactly so that previous slide mentioned

Conway's Game of Life and this is an

example of something that's cheering

complete so this is something that uh

every step it it goes through

generations every step and each of those

steps the state of one cell is

determined by it the state of the

previous state of the cells around it so

it's a little bit like an actual living

cell if the cell has no uh living cells

around it it's going to die of

loneliness if it has too many cells

around it that are alive it's going to

dive overcrowding but if it's got the

right number of cells uh around an empty

space the cell will actually uh come to

life so this is an example of something

that can kind of go on for ever

generating these

patterns very kind of common example of

something that's uh used as an

equivalent of cheing completeness yeah

so if this is an example of something

that's cheing complete is it something

that we can write in

ebpf and one reason why you might

think I don't think it can is because of

this thing the ebpf

verifier so when you load a program into

the kernel the verifier is going to run

o over this program and analyze the

program and establish whether it's safe

to run and one of the things I would

normally say when I talk about whether

it's safe to run I'll say it's not going

to crash the kernel but I would also say

that it's going to run to completion we

might pause on that

later you want to dive intoit more what

what it's really doing yeah yeah so I I

think um users of ebpf or or even people

that are in the stack you know they

might think about the verifier as this

kind of thing that is there stopping you

from writing interesting programs

perhaps um as a konel developer though I

want to kind of lay out what do we as

developers of the Kel that we want to

ensure that the Colonel's safe to run

and all this kind what do we think the

verifier is doing right and and

specifically this is kind of my criteria

um you know the first thing we always

want to do when we load a BPF program in

to the colonels we want to make sure

that we can only read memory that is

allowed by that program and by your

permissions that you have when you load

that program and and BPF has a whole

series of places you can hook in the

kernel you can hook kernel functions you

can hook the networking stack um you can

even hook user space and and depending

on where you hook you'll have different

permissions on what kind of memory you

can read right so we want to make sure

that your program is only reading memory

it's supposed to and we definitely 100%

want to make sure that you're only

writing to memory that you're allowed to

you know want to make sure that you're

not writing to random spots in memory

you're only writing to you know valid

locations in your vpf program um the

next one is we want to make sure that

the control flow is well formed and that

it's valid and in bounds we don't want

you to jump your BPF program off into

the kernel somewhere and run an

arbitrary code and crash the kernel

right don't want to make sure that

happens and I think the fourth Point

here is really interesting because we'

we've actually evolved this over time

it's like what does it mean for ebfp

programs to be bounded in the kernel and

if you think where the original BPF that

meant the program ran to completion but

over time we've sort of evolved BPF now

BPF programs know how to sleep we know

how to do callbacks and iterations we

know how to keep a program running and

we'll go into that in a little bit um

and really what we want to verify in the

verifier is that we don't get a Program

stuck on the CPU so long that it looks

and feels like the system is hung right

and for kernel developers or or people

that are really system level programmers

I think this is a sort of intuitive

concept because you've been writing in

this low level if you're if you're a

higher up in the stack and maybe an

application programmer or um kind of a

distrib systems programmer somewhere in

there it may be foreign because um you

don't have to worry about this stuff

because the OS is doing all the context

switching right you you never in never

write an application in user space C++ C

go whatever where you go like at this

point I need to make sure that I release

the CPU right because the the OS just

deals with that for you right once you

get yourself into the kernel into you

kind of the deep system stuff you really

need to um tell the OS when it's okay to

um to let your program off the CP view

right the reason we do this is really if

you're in an interrupt context if you're

uh trying to write some kind of locking

and stuff like this we can't just you

know release it um at any arbitrary

Point um and then the last thing we want

to do with the verifier is we want to

make sure like locks and references are

all accounted for and that that's sort

of a an accounting detail but it's

important if you're going to like

reference a socket or reference a file

um walk a path you want to make sure

that those things exist when you're

doing that and that's just a um kind of

a correctness property so th those are

the things um you know that

that I think about when I'm working on

the verifier or improving the verifier

um over time

so and I think as we'll see later this

idea that ebpf continues to evolve and

continues to improve over time because

of the work that people like John are

doing this is why we're able to perhaps

do more with ebpf than you might imagine

because it continues to improve yeah so

all right John can we run Game of Life

in ebpf all right we're gonna give it a

try here because we like to do demos so

what you see here is um this is just

basically tetragon um but we added a new

sensor to tetragon called Game of Life

all right and so we're going to kick it

off um what it did here is it just

iterate created the very similar to what

Liz just showed you with that um web app

is it created a a

map filled in some cells with some

random pattern and we could load it from

a BPF map or something if we wanted to

but for the for the demo it's just a

random map

and now it's running and it's there it

goes all right so while that's running

Let's uh dig into how you were able to

get this to to work

yeah so this is just a brief I'm not

going to go into all the details of

everything here but this is basically

the core um code that is running right

now in the background and we'll get back

to it but the important pieces I do want

to call out is that very far left for

you guys is do game this is our core

game logic for life here and you can see

kind of three things we copy the cell

map it's basically taking the Old State

copying it into the new state so that we

have a good State for the next iteration

we do the next iteration which is all of

those rules are run that Liz was talking

about you check to see if your um if

your neighbors are alive Al or you

create a new cell or you remove a new

cell so that's the sort of the business

logic of this program and then we send

an update all right so what the update

it does is it sends that map State up to

user space you know unfortunately you

know I don't have a a BPF Graphics

Library yet so you know if somebody's

interested and they have like a GPU

Shader from BPF or something to come

talk to me let's make it happen but I

couldn't do it from BPF so I had to send

the map up to user space slightly

disappointing but doesn't really impact

you know the the logic of the game right

so the user space is actually taking

that BPF map and just drawing it on the

screen there is what you were seeing and

then the last piece is this run next

which just reruns the business log but

goes back to the beginning of du game

and runs it again every two seconds all

right and we'll talk about how that

happens other slides you can get to in

in your in your spare time later if

you're

interested so you may have heard that

ebpf programs are limited to only a

small number of instructions so maybe

let's dive into that yeah um this is you

know what what's really interesting here

is you know when we first landed BPF and

414 um back in like 2015 kind of the

first

iterations um we really started at 4,000

instructions and like why did we pick

4,000 instructions because it seemed

like a good number it's you know Colonel

developers like you know like numbers

with k at the end of them 496 sounds

good right

um that was interesting but we very

quickly realized that you know 496 is

kind of small right you might want more

instructions um and the the other thing

I think is important

why do we have to have a limit was not

primarily because um we need the program

to terminate in some finite time and we

decided 4K instructions was a good

amount of time but I think it's really

if you think about it we have to verify

that program and we really want to make

sure that when you load a program into

the kernel it takes some reasonable

amount of time you know less than a

second right if you loaded a 20 million

um line code program perhaps maybe it

would take 20 seconds and you know we

kind of thought well a long time right

so keep it bounded but at one some point

we decided that 4K was just too limiting

people want to write things that are

bigger than 4K and so in 54 Colonels we

went to 1 million um and and kind of

beefed up that allow allowable

limit um just kind of follow up you

might wonder like that's interesting

John but give me some kind of reference

you know like how many instructions

really is a million instructions and so

what I did is just this is kind of a

napkin approach you know know go look at

a few programs I had on my laptop of

course I had Doom right got to have Doom

there so you know I took a look it's you

know less than a,k instructions for a

for a basic kind of Doom clone if you

think of the like the original Dune and

then I had Envoy on my system because um

you know we work on celium and tetragon

and that uses Envoy as a kind of one one

option for doing proxies I took a look

15 million instructions and then clang

uh you know compiler stuff 26 million

instructions so that's kind of where

we're at

um you know you might come back and say

okay Doom is only maybe less than a

thousand 100,000 instructions but John

it might have some Loops how are you

going to deal with that right and then

stay tuned we'll talk about how to get

these kind of much larger cases going

forward so there's also this idea that

and it was true in the early days that

you couldn't do loops in BPF because you

were only able to jump forwards in a

program yeah so this was something that

we put in and it not because necessarily

PF like um folks that were working on

BPF thought hey Loops are somehow bad it

was purely like the verifier needs to

somehow ensure that this isn't going to

run forever right um and that was the

early days and so we simplest thing to

do is look at your program and ensure

that you don't have any jumps to code

that you've already been to right we

want to make sure that the program

control flow is a d basically a graph

and no Loops in that graph um so early

PR programming what we did is we did

something called called an unroll

basically you tell the compiler I'm not

allowed to do loops please just cut and

paste this code right it'd be very

similar to a developer just doing like

cut cut paste their Loop many many times

you kind of do that but you can see very

quickly that you're not going to be able

to get things to to run very long right

especially in 4k instructions but even

with a million instructions you know

you're kind of only so much can be

done yeah so clearly not a not an

approximation of forever right when

you're kind of in this space um the next

piece here is we

have we as we were evolving BPF this

became a very obvious sort of limitation

right the fact that the limitation in

two way two ways I'd say is one is it

makes it hard to write programs and two

it makes it hard for the compiler to

compile things you know optimize your

code basically so we added Loops this is

this allows the verifier to look at a

loop and verify that that Loop is going

to terminate um and then we added an

actually helper call for cases where you

want the BPF program to do a loop but

the compiler human and verifier are

having trouble all agreeing on how that

Loop is going to look like actually and

if you think about the assmbly code and

object code when it's loaded in the

compiler and once you have this you can

write statements like that at the bottom

where you have a four Loop H here is a

um is a you know kind of a pound Define

or a global variable that tells you the

max loop size and and the verifier is

perfectly happy with this um today so

that's we're allowed to Loop but we're

not allowed to Loop indefinitely at this

point right correct yep that's why we

have that Max there and so if you just

kind of take a layout of what kernels

these were added in you know you 414 and

419 we had our kind of basic cut and

paste model 54 we added Loop support so

the colonel could figure out how if

these Loops are going to terminate 515

made a bunch of technical improvements

to that so the verifier got much better

at finding these things and then 61 we

added this um call that you can do to

basically you as the human can tell the

verifier through the compiler that hey

this is going to be a loop please take a

look at it and make sure that it

terminates yeah and I just wanted to

call this out because um you know we

have people working on this the the BPF

site is just evolving continuously so

there's another type of loop coming

we're not going to talk about it here

find me afterwards if you want but even

even uh more improved version of looping

is is on the way

so Okay so we've

got a limit to how many uh instructions

but it's quite a big limit we can Loop

yeah yeah

what about these big program what about

these big programs can we do Envoy or

Clan could you write in theory a $15

million 15 million instruction ppf

program uh that would be great right um

the thing that is interesting and I I

think it's it's a choice we made very

early on in the BPF um kind of verifier

is we call these things subprograms and

we said subprograms must terminate

roughly equivalent a subprogram is is

really roughly equivalent to a function

and so the limits are actually on the

functions not on the entire program and

as sort of a back a napkin you can do

like um 31 function calls or 31

subprograms is what a BPF programmer

would say and really that gets you up to

you know 15 million without much trouble

31 million actually with 31 calls right

um You can actually do more in the

kernel depending on where you're

actually being where you're hooking but

sort of as a rough estimate we can

pretty easily get to 31 million

instructions which is a really big

program right you can all of clay is not

even 31 million instructions so give you

some words of warning you know this uh

the verifier is getting better and

better and better and people are working

on this and making it usable for humans

um to write code right um but really the

trick right now is the human's going to

write the code the compiler's going to

compile it and the verifier is going to

verify it and they all need to kind of

be in sync and understand what's going

on right um to get the kind of something

into the kernel right and this is why

it's just not as easy as it should be

you know the team is working on this um

I fully predict that in 5 years 10 years

something like that BPF will be easier

and easier to write and and kind of open

up that that uh number of people that

are you know willing to dive in and

really get into the BPF

space um and then the last thing you

know if you do go this way and you maybe

have the right mentality it might be

addictive you might try to write things

like Tetris and doom and Quake and Game

of Life in the kernel um you know the

utility of that is um kind of great for

cubec con talks right I'm not sure you

know like is surveillant Enterprise will

have a or you know yeah we we're doing

the I surveillant Enterprise launch of

Game of Life yeah exactly you heard it

here first yeah yep um so Game of Life

you know it goes on forever so what

about this idea of not

terminating yeah so this is an

interesting one right so like if you

want something to run forever I've told

you there's a maximum instructions

you're like well certainly you're going

to run out of those instructions at some

point right and this really goes back to

this kind of carefully worded um point

in the verifier where we said well it's

not just that we want the program to

terminate it's that we want to release

the CPU that we're on we don't want to

lock a CPU down and like we said early

on early kernel versions that meant the

program terminated there was no

exceptions to that the program was done

um but when we evolved the um kind of

Kernel and BPF and the verifier and the

compiler as well like all these pieces

are moving moving forward we developed

these kind of other ideas of kind of Rel

other ways to release the CPU um one of

them was an iterator this is a program

that can like run over every file in a

directory right it doesn't matter how

many files are in there and it'll

basically ensure that the CPU doesn't

get locked up right it's been iterating

for too long it'll release let something

else run the scheduler will come back

and say okay BPF program you can run

again um we've allowed programs to sleep

now so this is really really useful if

like you're trying to read some memory

and it gets the page fold so the memory

is not in in the page cache so you can't

just read it because it's not there um

and so you have to do something called a

page fault which requires sleeping

anyways it's it's in the details but

super important if you're be program

tries to read some memory and it you

know you don't want to get a fault we

added timer call backs this is basically

a way to say hey I've done something

useful um I'm going to release my

program let the OS do something and then

just call me back at some time you can

even set this to zero which just looks

like hey I'm done um but you know I'm

safe to release but call me as soon as

you can again

so and I guess the last thing is you

know how can an ebpf program allocate

some amount of memory yeah so the um

there's really a bunch of different ways

we can do this but the most obvious one

is that we have these array maps and the

maps are basically memory blocks that

the program can all ate or user space

can allocate and then give to the

program and um these can be actually

quite large the basic limit of these is

you know like whatever the user space is

allowed to allocate based on its cgroup

if it's not in a cgroup they can be more

or less unbounded and kind of related to

system memory and all this kind of stuff

we have some limits but those are tied

to like how much RAM you have in your

system right same type of thing that a

user space application has you can't

allocate more memory than you know your

system has available right so so we

could pot entally have a pretty big

absolutely uh sort of field of play for

our game yeah yeah and you know even you

know four gigabytes we've sort of

optimized this case in newer kernels to

allow applications up to four gigabytes

of virtual memory um amazing stuff so

with all of these component pieces put

together we've really avoided the

limitations right we don't have uh we

well we don't really have an effective

limit on how many instructions we don't

have an effective limit on how many time

times we can call a loop effectively

because we can use these timer call

backs shall we see if it's still running

what do you think is it still

running yeah so see how good I I am at

programming right if it's still running

or if it's still going so here you go

looks like it's still up and running um

you can see it's still doing its thing

um and if you go back to kind of some of

the details um I think we can talk for a

couple seconds here basically if you

think about that first program that was

doing in that due game you know what we

see here is it does a copy of the last

state into the new state runs the logic

that's what moves all these kind of

fields around on the screen right and

then um sends it up to user space and

then this is the user space just kind of

printing the dots you know one's on zero

is off printing the dots on the screen

and you can see it kind of it keeps

evolving over time every every two

seconds is the limit um just so that you

all can see it

um so there it goes and it's it's still

going you told me a pretty interesting

thing about like the fact that you can't

predict the future State yeah yeah one

of the neat things about like game and

life and and a lot of terrain machines

in general is that this thing will um

you you if you want to know what the

thousandth iteration is or the 10,000th

iteration of this there's no shortcut to

that from a from a kind of a

mathematical side you can't say tell me

I'm not going to actually run this thing

10,000 steps let me just run it two

times and then tell me what the answer

is is so like it's interesting in that

sense is you have to you really have to

run this thing to figure it out right um

and uh we could rerun it and see what

pattern we get you know sometimes you'll

get interesting patterns this one is

still running um since it's a random

pattern we could have been unlucky right

and had like the screen be black when we

turned this on but um you know but it

didn't it work Round of Applause for a

live demo please

great perfect yep go back

amazing so we have Game of Life in ebpf

which is really a good demonstration

that you can today do arbitrarily

complex tasks with

ebpf so those statements about things

like can you do layer sing protocol

passing in

ebpf absolutely it's possible doesn't

necessarily mean that all these things

have been implemented yet but it is

possible to do you know pretty much any

processing maybe not necessarily to do

the display part yet but the actual

computation part the processing part and

and I would say the key is there not yet

I mean I I think the really interesting

thing or one takeaway I think from this

talk is like you know there's a bunch of

folks in the kernel in the compiler

community and you know we're working to

make BPF better and make BPF improve BPF

so if you have a use case and if your

use case is graphics on GPU for your

cubec con talk come talk talk to me it

would be interesting you know like so

these things can be we can make them

happen right you know in 10 years ago

when we got started we had like this

very small scope and you know it just

keeps growing and growing the community

is getting bigger you know it's kind of

its own sub Community with ebpf but um

you know I I think definitely if you

have a use case and you think there's

value in it you going come talk to us

and we can we can figure out what it

what it takes to make it happen and I

think when we think about selum you know

selum is using ebpf for lot of network

based processing and you know our vision

is to push more and more of that

processing into the kernel so yeah

there's tons of networking related

processing happening in user space today

where I'm not going to tell you that you

know we're writing it and it's going to

be released next week that's not what

I'm trying to say but I'm trying to say

the vision is that more and more of this

processing will be happening in the

kernel yeah and I I mean I I would also

just say absolutely the same for I I

work on tetron mostly these days it's

the same thing early we started on

tetron a couple years back you know

there was missing features we added the

features and then you you figure out how

to bridge the gap in the interum but if

you look forward in two three years you

can see like all the cloud providers

will have that kernel all the uh

distributions will have that kernel and

you can start to use that functionality

right and that's I think um a really

powerful kind of kind of point of the

open source effort

here so I hope that's convinced you that

ebpf is well probably more powerful than

you

thought if you want to learn more about

ebpf we have uh some books that you can

download from is surveillent.com there's

also a ton of really great Labs on the

website there is the ebpf doio site

which has loads of information about

ebpf itself and lots of projects that

are being uh built using it uh we're

also doing uh well I'm going to be

signing some books at the I surveillant

booth at 1:00 today so if you want to

come around and pick up a copy of the

book then uh please say hello and with

that I hope you have a wonderful wrap po

to the end of your

keepon