We write our applications in ebpf: A Tale From a Telekom Operator - Nick Zavaritsky
Summary
TLDRNick from mnii explains the company's use of eBPF for efficient data packet processing in smart devices, such as smart mouse traps. They transitioned from using GPDK to eBPF, which aggregates GTP traffic into a single GRE tunnel, enhancing performance and handling complex scenarios. eBPF's ability to run multiple packet processing apps simultaneously is a game-changer, offering high throughput and flexibility. Challenges with state management and Kubernetes integration are addressed, showcasing eBPF's potential in modern network applications.
Takeaways
- 🐭 Nick's company, mnii, uses smart mouse traps that send push notifications when a mouse is caught, highlighting the importance of always-online connectivity.
- 📡 The smart traps are independent of Wi-Fi and use mui cement side technology for data shipping, emphasizing the shift from gpdk to ebpf for efficiency.
- 🚀 Easier data path is achieved by integrating AWS, allowing device traffic to enter the customer VPC directly without public internet exposure.
- 🔄 Transitioning from GTP to GRE tunnels is simplified using TC programs with BPF SKB get tunnel key helper, showcasing the power of BPF for packet processing.
- 🌐 The script discusses the complexity of handling multiple sessions and tunnels, where BPF's ability to aggregate traffic and manage state is crucial.
- 🛠️ BPF's flexibility is highlighted by its ability to run multiple packet processing applications side by side, a significant advantage over other methods.
- 📚 The script acknowledges the complexity of signaling in cell networks but chooses to focus on the data path for simplicity.
- 🔄 BPF maps are praised for their powerful API, complete type information, and the ability to build generic tools for state management.
- 💾 The script suggests that BPF can be used to populate maps from database tables, indicating its versatility in data handling.
- 🔒 Security is implied through the use of AWS integration and the handling of traffic within VPCs, ensuring data is kept private and secure.
- 🛠️ The script concludes that while DPDK is faster, BPF's ability to run multiple applications concurrently and its integration with Kubernetes is a game-changer.
Q & A
What is the primary function of the smart mouse traps mentioned in the script?
-The smart mouse traps are designed to trap mice, and once a mouse is trapped, they send a push notification.
Why does the smart mouse trap not depend on Wi-Fi for connectivity?
-The smart mouse traps are always online and do not rely on Wi-Fi because they have built-in cellular connectivity.
What is the significance of using eBPF (Extended Berkeley Packet Filter) in the context of the script?
-eBPF is seen as a game-changer because it allows for running multiple packet processing applications side by side, which is crucial for handling the complexity of the service described.
How does the data path work for the smart mouse traps once a device is online and authenticated?
-The data path involves a GTP tunnel that is terminated on the right-hand side, with traffic entering the customer's VPC through AWS integration without traveling over the public internet.
What is the role of AWS Transit Gateway in the data path described?
-AWS Transit Gateway understands GRE (Generic Routing Encapsulation) and can match the tunnel to the customer VPC, facilitating the transfer of traffic.
Why is there a need to convert between GTP tunnels and GRE tunnels?
-The conversion is necessary because the system has over 1 million GTP tunnels but uses a single GRE tunnel in collect mode to aggregate all GTP traffic, which is more efficient.
How does the TC (Traffic Control) program use eBPF to process the tunnels?
-The TC program uses eBPF to aggregate GTP traffic into a single tunnel, match a tunnel to the customer, and redirect packets into a GRE device, all while handling complex corner cases like packet reassembly and fragmentation.
What is the advantage of using XDP (eXpress Data Path) in the packet processing?
-XDP allows for fast packet processing at the earliest point in the network stack, matching a tunnel to the customer and finding the target GRE tunnel using shared eBPF maps.
Why is eBPF considered more flexible than DPDK (Data Plane Development Kit) in the context of the script?
-eBPF is more flexible because it allows running multiple packet processing applications side by side, whereas DPDK is limited to a single task per CPU core.
How does the script's author propose to handle the challenges of state management and software updates in the context of eBPF?
-The author suggests that eBPF maps, with their powerful APIs, can be used to build generic tools for dumping and restoring state, allowing for state extraction from a running instance and injection into a new one.
What challenges does the author face when trying to run eBPF-based applications in Kubernetes?
-The challenges include the incompatibility of eBPF with virtual interfaces and the need for a secondary interface with a custom CNI (Container Network Interface) to ensure packets are processed as needed by the multiple packet processing applications.
Outlines
🐭 Smart Mouse Trap Connectivity
Nick from mnii introduces a smart mouse trap as a metaphor for their IoT connectivity service. The trap, once triggered, sends a push notification without relying on Wi-Fi, using mui cement side technology. The company's transition from gpdk to ebpf is highlighted as a game-changer. The script explains the data path for IoT devices, emphasizing the simplicity of data passing through a GTP tunnel into the customer's VPC via AWS integration. The use of a TC program with BPF to aggregate GTP traffic into a single GRE tunnel is detailed, along with the handling of complex corner cases like packet reassembly and fragmentation. The script also touches on the broader challenges of managing multiple GTP tunnels and the benefits of using ebpf for stateful operations and high-performance packet processing.
🚀 High-Performance Packet Processing with eBPF and DPDK
The script contrasts eBPF with DPDK, highlighting eBPF's ability to run multiple packet processing applications side by side, which is a significant advantage over DPDK's single-task focus. While acknowledging DPDK's speed and flexibility, the script points out its limitations with polling and scalability. The discussion then shifts to the challenges of running packet processing applications in Kubernetes, where eBPF's integration is not straightforward due to its interaction with virtual interfaces and network plugins. The script concludes with the necessity of a custom CNI for eBPF applications in Kubernetes and the ongoing work to build a solution. The video ends with a summary of the benefits of using eBPF for smart device connectivity, the preference for microservices over monoliths, and the cool factor of running multiple packet processing apps concurrently.
Mindmap
Keywords
💡Smart Mouse Traps
💡M-NO (Mobile Network Operator)
💡GTP (GPRS Tunneling Protocol)
💡AWS Integration
💡GRE (Generic Routing Encapsulation)
💡BPF (Berkeley Packet Filter)
💡XDP (eXpress Data Path)
💡Collect Mode
💡Stateful Processing
💡Microservices Architecture
💡Kubernetes
Highlights
Nick works for a company called mnii and uses a smart mouse trap example to explain their work.
Smart mouse traps send push notifications when a mouse is caught, using mui cement side for connectivity.
The company is transitioning from gpdk to ebpf, which is seen as a game-changer.
Data packets are shipped using IP, with GTP tunnels terminated on the right-hand side.
AWS integration allows device traffic to enter the customer VPC without using the public internet.
Traffic is wrapped in GRE for compatibility with AWS Transit Gateway.
Over 1 million GTP tunnels are managed in a busy region, aggregated into a single GRE tunnel.
A TC program uses BPF SKB get tunnel key helper to recover tunnel information.
BPF maps are used to match tunnels to customers and find target GRE tunnels.
Packets are redirected into a GRE device operating in collect mode.
BPF handles complex corner cases like packet reassembly and fragmentation.
BPF is used for high-performance packet processing, with XDP programs for fast path matching.
BPF allows running multiple packet processing apps side by side.
BPF maps have a powerful API for populating and reading content with type information.
State can be extracted from a running instance and injected into a new one using BPF.
DPDK is faster than BPF but requires dedicating a CPU core to a single task.
BPF's ability to run multiple apps side by side is a significant advantage over DPDK.
BPF works well with Kubernetes, but some adjustments are needed for packet processing.
A secondary interface with a custom CNI is provisioned to meet packet processing needs.
The company is taking smart devices online using isdp and TC, with a focus on scalability.
The conclusion emphasizes the importance of BPF for running multiple packet processing apps in parallel.
Transcripts
[Music]
hi I'm Nick I work for a company called
mnii I'd like to use a cute example to
explain what we are doing I have a
customer that makes Smart mouse traps
once the mouse is trapped a push
notification is sent Mouse trp is always
online and it doesn't depend on your
Wi-Fi for connectivity because it has
mui cement side we don't offer voice
only data shipping IP packets is an
important part of our business we relied
on gpdk in the past but now we are
transitioning to ebpf today I want to
tell you why we see ebpf as a game
changer I'd like to acknowledge that
signaling in cell networks is a complex
topic I will conveniently ignore
signaling today once a device is online
and authenticated the data pass is
rather simple it is a GTP tunnel we have
to terminate on the right hand side we
have AWS integration device traffic gets
right into the customer VPC without
traveling over the public internet
actually it enters our VPC first the
traffic is wrapped in GRE because AWS
Transit Gateway understands GRE and it
can match the tunnel to the customer VPC
where appearing withs we had to convert
between GTP tunnels on the left hand
side and GR on the right hand side we
have over 1 million GTP tunnels in a
busy region but relatively few GRE
tunnels clearly you can can't have a
million tunnels in Linux instead a
single tunnel in so-called collect mode
Aggregates all GTP traffic a TC program
uses BPF SKB get tunnel key helper to
recover information about a tunnel the
program uses BPF maps to match a tunnel
to the customer and to find out the
target GRE tunnel finally the program
updates the tunnel key and redirects the
packet into a GRE device also operating
in collect mode it's reasonably fast
better yet it handles all complex Corner
cases such as packet reassembly and
fragmentation and sending icmp
fragmentation needed
messages Corner cases do happen but for
the majority of packets conversion
between GTP and gr is a simple as
changing a few headers in the packet
buffer we use XDP program was a fast PA
it match a tunnel to the customer and
finds out the target G tunnel using BPF
Maps shared with DC program so
apparently BPF is quite handy and
Powerful however we consider it the
Breakthrough for a different reason
let's have a look at the broader picture
there are some complications on the left
a device can open several sessions
simultaneously it results in multiple
GTP tunnels to be used for different
traffic classes on the right it is
typically a few GRE tunnels per customer
for improved throughput and redundancy
some customers do their own tunneling so
that packets can carry additional
metadata such as an Ino ID and in
between we have a nut and a firewall and
flow logs in the kitchen sink due to
overall complexity we prefer to do
several apps instead of a single
monolith so we have multiple apps to
operate and not just multiple apps but
multiple instances of every app it would
be nice if you could run several apps
side by side on a single server another
challeng is State product evolves and
requirements change therefore we have to
push updates of but the tel mightly for
days either we keep old versions of the
software out until it drains or we
impact customers it would be nice if you
could extract State out of a running
instance and somehow inject it into a
new one
with mpf it is finally possible to run
multiple packet processing apps side by
side that's why we write our
applications in mpf or more precisely
the packet processing part we use goang
for user space part we are H ebpf
users we manage our f with kubernets I
have to acknowledge that ebpf maps are
accidentally awesome they are equipped
with powerful API to populate and to
read back the content speak speaking of
content a complete type information is
available it is not OP pack bites you
are looking at but rather structures
with named fields of known type it is
possible to build a generic tool to dump
and restore State performing conversion
that data out changed with sufficient
effort streaming replication can be
achieved we can also populate a ebpf map
from a database table in a generic way
I'd love to dedicate more time to ebpf
Maps but unfortunately I
hand speaking about high performance
packet processing it's hard to ignore
the pdk it has been around for a while
and it is quite popular the idea is to
remove a network interface from the
kernel and to drive it from user space
pcket buffers are shared with user space
via memory mapping similarly receiving
transmitting are exposed application PS
receiving in a busy Loop to find out if
there is an incoming
packet I have to acknowledge that the in
general is faster than ebpf a modern CPU
has many special purpose instructions
such as AVX which ebpf lags when using
dpdk a CPU core is typically dedicated
to a single task hence CPU cach heat
rates are higher dpdk is also more
flexible since it is just regular user
space however dpdk means to BU the
polling as soon as there are more
threats and CPU cores everything falls
apart VIs BPF it is possible to run
multiple packet processing apps side by
side easily and it is
huge finally let's talk about running in
kubernets an application relying on isdp
works just fine in kubernets except it
doesn't an X Program won't see any
packets coming in it doesn't work with
virtual internet P it probably doesn't
work with net kit
either let's follow the path of an
incoming packet it's CC program attached
to e z observes a packet and redirects
it into vf1 normally a redirect would
put a packet on an internal queue inside
which is an pair and packet processing
would end eventually the Kel will find
out that there is a packet pending the
packet will reemerge from via T passing
through hdp andc layers is expected but
there is an important optimization
happening instead of a plane redirect
BBF redirect pi done the packet doesn't
go into cure the packet processing
doesn't end instead it performs another
iteration using grass interface set to
W2 since there is no waiting on the que
the latency improves several Corners are
cut and hdp is
skipped however for our packet
processing needs we actually want the
packet to get on a c consider this they
have many packet processing apps each
with their own ebpf blobs accumulating
multiple packets and processing the
batch together is more effective from
CPU cach perspective it also like to
dispatch packets to ports in DP since
going through TC to hit XDP makes little
sense apparently packet processing apps
have different needs compared to regular
container work claws since we don't see
a way to unify conflicting requirements
we provision a secondary interface with
a custom
cni it's time for a conclusion we are
taking smart devices online using isdp
and TC the service is complex therefore
we are doing Cals instead of a monolith
is the BPF it is finally possible to run
multiple packet processing apps side by
side is really cool however to make it
work in kubernets a proposed build C is
necessary we are currently working on
one
[Music]
[Applause]
[Music]
Weitere ähnliche Videos ansehen
eBPF’s Abilities and Limitations: The Truth - Liz Rice & John Fastabend, Isovalent
Looking Ahead: the eBPF Innovation Roadmap - Thomas Graf
Quality of Service (QoS) PART-2 Explained in Hindi l Embedded and Real time Operating System Course
SMT 2-6 Sniffing
What is Ethereum? A Beginner's Explanation in Plain English
Network Virtualization Simplified
5.0 / 5 (0 votes)