We write our applications in ebpf: A Tale From a Telekom Operator - Nick Zavaritsky

Summary

Nick from mnii explains the company's use of eBPF for efficient data packet processing in smart devices, such as smart mouse traps. They transitioned from using GPDK to eBPF, which aggregates GTP traffic into a single GRE tunnel, enhancing performance and handling complex scenarios. eBPF's ability to run multiple packet processing apps simultaneously is a game-changer, offering high throughput and flexibility. Challenges with state management and Kubernetes integration are addressed, showcasing eBPF's potential in modern network applications.

Takeaways

• 🐭 Nick's company, mnii, uses smart mouse traps that send push notifications when a mouse is caught, highlighting the importance of always-online connectivity.
• 📡 The smart traps are independent of Wi-Fi and use mui cement side technology for data shipping, emphasizing the shift from gpdk to ebpf for efficiency.
• 🚀 Easier data path is achieved by integrating AWS, allowing device traffic to enter the customer VPC directly without public internet exposure.
• 🔄 Transitioning from GTP to GRE tunnels is simplified using TC programs with BPF SKB get tunnel key helper, showcasing the power of BPF for packet processing.
• 🌐 The script discusses the complexity of handling multiple sessions and tunnels, where BPF's ability to aggregate traffic and manage state is crucial.
• 🛠️ BPF's flexibility is highlighted by its ability to run multiple packet processing applications side by side, a significant advantage over other methods.
• 📚 The script acknowledges the complexity of signaling in cell networks but chooses to focus on the data path for simplicity.
• 🔄 BPF maps are praised for their powerful API, complete type information, and the ability to build generic tools for state management.
• 💾 The script suggests that BPF can be used to populate maps from database tables, indicating its versatility in data handling.
• 🔒 Security is implied through the use of AWS integration and the handling of traffic within VPCs, ensuring data is kept private and secure.
• 🛠️ The script concludes that while DPDK is faster, BPF's ability to run multiple applications concurrently and its integration with Kubernetes is a game-changer.

Q & A

What is the primary function of the smart mouse traps mentioned in the script?

The smart mouse traps are designed to trap mice, and once a mouse is trapped, they send a push notification.

Why does the smart mouse trap not depend on Wi-Fi for connectivity?

The smart mouse traps are always online and do not rely on Wi-Fi because they have built-in cellular connectivity.

What is the significance of using eBPF (Extended Berkeley Packet Filter) in the context of the script?

eBPF is seen as a game-changer because it allows for running multiple packet processing applications side by side, which is crucial for handling the complexity of the service described.

How does the data path work for the smart mouse traps once a device is online and authenticated?

The data path involves a GTP tunnel that is terminated on the right-hand side, with traffic entering the customer's VPC through AWS integration without traveling over the public internet.

What is the role of AWS Transit Gateway in the data path described?

AWS Transit Gateway understands GRE (Generic Routing Encapsulation) and can match the tunnel to the customer VPC, facilitating the transfer of traffic.

Why is there a need to convert between GTP tunnels and GRE tunnels?

The conversion is necessary because the system has over 1 million GTP tunnels but uses a single GRE tunnel in collect mode to aggregate all GTP traffic, which is more efficient.

How does the TC (Traffic Control) program use eBPF to process the tunnels?

The TC program uses eBPF to aggregate GTP traffic into a single tunnel, match a tunnel to the customer, and redirect packets into a GRE device, all while handling complex corner cases like packet reassembly and fragmentation.

What is the advantage of using XDP (eXpress Data Path) in the packet processing?

XDP allows for fast packet processing at the earliest point in the network stack, matching a tunnel to the customer and finding the target GRE tunnel using shared eBPF maps.

Why is eBPF considered more flexible than DPDK (Data Plane Development Kit) in the context of the script?

eBPF is more flexible because it allows running multiple packet processing applications side by side, whereas DPDK is limited to a single task per CPU core.

How does the script's author propose to handle the challenges of state management and software updates in the context of eBPF?

The author suggests that eBPF maps, with their powerful APIs, can be used to build generic tools for dumping and restoring state, allowing for state extraction from a running instance and injection into a new one.

What challenges does the author face when trying to run eBPF-based applications in Kubernetes?

The challenges include the incompatibility of eBPF with virtual interfaces and the need for a secondary interface with a custom CNI (Container Network Interface) to ensure packets are processed as needed by the multiple packet processing applications.

Outlines

00:00

🐭 Smart Mouse Trap Connectivity

Nick from mnii introduces a smart mouse trap as a metaphor for their IoT connectivity service. The trap, once triggered, sends a push notification without relying on Wi-Fi, using mui cement side technology. The company's transition from gpdk to ebpf is highlighted as a game-changer. The script explains the data path for IoT devices, emphasizing the simplicity of data passing through a GTP tunnel into the customer's VPC via AWS integration. The use of a TC program with BPF to aggregate GTP traffic into a single GRE tunnel is detailed, along with the handling of complex corner cases like packet reassembly and fragmentation. The script also touches on the broader challenges of managing multiple GTP tunnels and the benefits of using ebpf for stateful operations and high-performance packet processing.

05:01

🚀 High-Performance Packet Processing with eBPF and DPDK

The script contrasts eBPF with DPDK, highlighting eBPF's ability to run multiple packet processing applications side by side, which is a significant advantage over DPDK's single-task focus. While acknowledging DPDK's speed and flexibility, the script points out its limitations with polling and scalability. The discussion then shifts to the challenges of running packet processing applications in Kubernetes, where eBPF's integration is not straightforward due to its interaction with virtual interfaces and network plugins. The script concludes with the necessity of a custom CNI for eBPF applications in Kubernetes and the ongoing work to build a solution. The video ends with a summary of the benefits of using eBPF for smart device connectivity, the preference for microservices over monoliths, and the cool factor of running multiple packet processing apps concurrently.

Keywords

💡Smart Mouse Traps

Smart mouse traps refer to a type of device that uses technology to capture mice. In the context of the video, these traps are equipped with connectivity that allows them to send push notifications once a mouse is caught. This example is used to illustrate the practical application of advanced networking technologies in everyday devices, showcasing the integration of IoT (Internet of Things) principles.

💡M-NO (Mobile Network Operator)

M-NO stands for Mobile Network Operator, which is a company that provides wireless communication services to mobile phone users. The script mentions M-NO in relation to the connectivity of smart devices, emphasizing the importance of reliable and robust network services for the operation of IoT devices like smart mouse traps.

💡GTP (GPRS Tunneling Protocol)

GTP is a network protocol used in mobile networks to transport data across a wireless network. In the video, GTP is mentioned as the protocol used for the data path in smart mouse traps, which must be terminated and translated into another protocol, GRE, for efficient data transmission within a networked environment.

💡AWS Integration

AWS, or Amazon Web Services, is a comprehensive and broadly adopted cloud platform. The script discusses how device traffic from smart mouse traps gets routed into a customer's VPC (Virtual Private Cloud) on AWS without going over the public internet, highlighting the security and efficiency of such integration.

💡GRE (Generic Routing Encapsulation)

GRE is a protocol used to encapsulate a variety of network layer protocols. The video explains that traffic from the smart mouse traps, which initially uses GTP, is wrapped in GRE for compatibility with AWS Transit Gateway, demonstrating the technical steps involved in data transmission across different network environments.

💡BPF (Berkeley Packet Filter)

BPF is a technology for executing packet processing programs in the kernel space, which is crucial for high-performance networking. The video discusses how BPF is used to manage the conversion between GTP and GRE tunnels efficiently, emphasizing its role in handling complex packet processing tasks.

💡XDP (eXpress Data Path)

XDP is a high-performance networking bypass framework in the Linux kernel. The script mentions using XDP for fast packet processing, indicating its importance in achieving low-latency network operations, which is critical for real-time applications like smart devices.

💡Collect Mode

Collect mode is a method of aggregating traffic from multiple sources into a single stream. The video describes how a single GTP tunnel in collect mode can aggregate all GTP traffic, which simplifies the management of a large number of tunnels and enhances network efficiency.

💡Stateful Processing

Stateful processing refers to the ability of a system to keep track of the state of network connections. The script touches on the challenges of managing stateful processing in evolving applications, where BPF provides a solution by allowing state to be extracted and injected into new instances.

💡Microservices Architecture

Microservices architecture is a design approach that structures applications as collections of loosely coupled services. The video discusses the preference for microservices over monolithic applications due to the complexity of managing smart devices, highlighting the scalability and flexibility of this architectural pattern.

💡Kubernetes

Kubernetes is an open-source platform designed to automate deploying, scaling, and operating application containers. The script mentions the use of Kubernetes to manage BPF applications, indicating the integration of advanced networking technologies with container orchestration systems for efficient management of cloud-native applications.

Highlights

Nick works for a company called mnii and uses a smart mouse trap example to explain their work.

Smart mouse traps send push notifications when a mouse is caught, using mui cement side for connectivity.

The company is transitioning from gpdk to ebpf, which is seen as a game-changer.

Data packets are shipped using IP, with GTP tunnels terminated on the right-hand side.

AWS integration allows device traffic to enter the customer VPC without using the public internet.

Traffic is wrapped in GRE for compatibility with AWS Transit Gateway.

Over 1 million GTP tunnels are managed in a busy region, aggregated into a single GRE tunnel.

A TC program uses BPF SKB get tunnel key helper to recover tunnel information.

BPF maps are used to match tunnels to customers and find target GRE tunnels.

Packets are redirected into a GRE device operating in collect mode.

BPF handles complex corner cases like packet reassembly and fragmentation.

BPF is used for high-performance packet processing, with XDP programs for fast path matching.

BPF allows running multiple packet processing apps side by side.

BPF maps have a powerful API for populating and reading content with type information.

State can be extracted from a running instance and injected into a new one using BPF.

DPDK is faster than BPF but requires dedicating a CPU core to a single task.

BPF's ability to run multiple apps side by side is a significant advantage over DPDK.

BPF works well with Kubernetes, but some adjustments are needed for packet processing.

A secondary interface with a custom CNI is provisioned to meet packet processing needs.

The company is taking smart devices online using isdp and TC, with a focus on scalability.

The conclusion emphasizes the importance of BPF for running multiple packet processing apps in parallel.

Transcripts

[Music]

hi I'm Nick I work for a company called

mnii I'd like to use a cute example to

explain what we are doing I have a

customer that makes Smart mouse traps

once the mouse is trapped a push

notification is sent Mouse trp is always

online and it doesn't depend on your

Wi-Fi for connectivity because it has

mui cement side we don't offer voice

only data shipping IP packets is an

important part of our business we relied

on gpdk in the past but now we are

transitioning to ebpf today I want to

tell you why we see ebpf as a game

changer I'd like to acknowledge that

signaling in cell networks is a complex

topic I will conveniently ignore

signaling today once a device is online

and authenticated the data pass is

rather simple it is a GTP tunnel we have

to terminate on the right hand side we

have AWS integration device traffic gets

right into the customer VPC without

traveling over the public internet

actually it enters our VPC first the

traffic is wrapped in GRE because AWS

Transit Gateway understands GRE and it

can match the tunnel to the customer VPC

where appearing withs we had to convert

between GTP tunnels on the left hand

side and GR on the right hand side we

have over 1 million GTP tunnels in a

busy region but relatively few GRE

tunnels clearly you can can't have a

million tunnels in Linux instead a

single tunnel in so-called collect mode

Aggregates all GTP traffic a TC program

uses BPF SKB get tunnel key helper to

recover information about a tunnel the

program uses BPF maps to match a tunnel

to the customer and to find out the

target GRE tunnel finally the program

updates the tunnel key and redirects the

packet into a GRE device also operating

in collect mode it's reasonably fast

better yet it handles all complex Corner

cases such as packet reassembly and

fragmentation and sending icmp

fragmentation needed

messages Corner cases do happen but for

the majority of packets conversion

between GTP and gr is a simple as

changing a few headers in the packet

buffer we use XDP program was a fast PA

it match a tunnel to the customer and

finds out the target G tunnel using BPF

Maps shared with DC program so

apparently BPF is quite handy and

Powerful however we consider it the

Breakthrough for a different reason

let's have a look at the broader picture

there are some complications on the left

a device can open several sessions

simultaneously it results in multiple

GTP tunnels to be used for different

traffic classes on the right it is

typically a few GRE tunnels per customer

for improved throughput and redundancy

some customers do their own tunneling so

that packets can carry additional

metadata such as an Ino ID and in

between we have a nut and a firewall and

flow logs in the kitchen sink due to

overall complexity we prefer to do

several apps instead of a single

monolith so we have multiple apps to

operate and not just multiple apps but

multiple instances of every app it would

be nice if you could run several apps

side by side on a single server another

challeng is State product evolves and

requirements change therefore we have to

push updates of but the tel mightly for

days either we keep old versions of the

software out until it drains or we

impact customers it would be nice if you

could extract State out of a running

instance and somehow inject it into a

new one

with mpf it is finally possible to run

multiple packet processing apps side by

side that's why we write our

applications in mpf or more precisely

the packet processing part we use goang

for user space part we are H ebpf

users we manage our f with kubernets I

have to acknowledge that ebpf maps are

accidentally awesome they are equipped

with powerful API to populate and to

read back the content speak speaking of

content a complete type information is

available it is not OP pack bites you

are looking at but rather structures

with named fields of known type it is

possible to build a generic tool to dump

and restore State performing conversion

that data out changed with sufficient

effort streaming replication can be

achieved we can also populate a ebpf map

from a database table in a generic way

I'd love to dedicate more time to ebpf

Maps but unfortunately I

hand speaking about high performance

packet processing it's hard to ignore

the pdk it has been around for a while

and it is quite popular the idea is to

remove a network interface from the

kernel and to drive it from user space

pcket buffers are shared with user space

via memory mapping similarly receiving

transmitting are exposed application PS

receiving in a busy Loop to find out if

there is an incoming

packet I have to acknowledge that the in

general is faster than ebpf a modern CPU

has many special purpose instructions

such as AVX which ebpf lags when using

dpdk a CPU core is typically dedicated

to a single task hence CPU cach heat

rates are higher dpdk is also more

flexible since it is just regular user

space however dpdk means to BU the

polling as soon as there are more

threats and CPU cores everything falls

apart VIs BPF it is possible to run

multiple packet processing apps side by

side easily and it is

huge finally let's talk about running in

kubernets an application relying on isdp

works just fine in kubernets except it

doesn't an X Program won't see any

packets coming in it doesn't work with

virtual internet P it probably doesn't

work with net kit

either let's follow the path of an

incoming packet it's CC program attached

to e z observes a packet and redirects

it into vf1 normally a redirect would

put a packet on an internal queue inside

which is an pair and packet processing

would end eventually the Kel will find

out that there is a packet pending the

packet will reemerge from via T passing

through hdp andc layers is expected but

there is an important optimization

happening instead of a plane redirect

BBF redirect pi done the packet doesn't

go into cure the packet processing

doesn't end instead it performs another

iteration using grass interface set to

W2 since there is no waiting on the que

the latency improves several Corners are

cut and hdp is

skipped however for our packet

processing needs we actually want the

packet to get on a c consider this they

have many packet processing apps each

with their own ebpf blobs accumulating

multiple packets and processing the

batch together is more effective from

CPU cach perspective it also like to

dispatch packets to ports in DP since

going through TC to hit XDP makes little

sense apparently packet processing apps

have different needs compared to regular

container work claws since we don't see

a way to unify conflicting requirements

we provision a secondary interface with

a custom

cni it's time for a conclusion we are

taking smart devices online using isdp

and TC the service is complex therefore

we are doing Cals instead of a monolith

is the BPF it is finally possible to run

multiple packet processing apps side by

side is really cool however to make it

work in kubernets a proposed build C is

necessary we are currently working on

one

[Music]

[Applause]

[Music]