FPGAs and low latency trading - Williston Hayes - Optiver - FPL2020
Summary
TLDRThis presentation explores the intersection of FPGAs and low-latency trading, explaining how FPGAs can significantly reduce the time it takes to process trading decisions. The speaker from Optiver, a global market-making firm, discusses the architecture of an FPGA-based trading system and the unique design approach they use, focusing on minimizing latency and maximizing performance. The talk also covers the real-time impact of FPGA design on trading strategies and the dynamic, innovative solutions required in this high-paced financial environment.
Takeaways
- π The speaker was initially unaware of the intersection between FPGAs and low-latency trading until a job offer from Optiver sparked interest.
- π¦ Optiver is a global market-making firm founded in 1986, trading various financial instruments with their own capital and focusing on improving market liquidity.
- π The transition from traditional trading pits to electronic trading in data centers has significantly impacted how trades are executed, with latency becoming a critical factor.
- π The importance of low latency in trading is highlighted by the 'first come, first serve' mechanism used by exchanges to handle simultaneous orders.
- π Limitations of software-based trading systems include latency penalties from traversing the NIC, PCI Express, CPU, and memory, which FPGAs can help mitigate.
- π οΈ FPGAs can reduce latency by handling trading logic close to the network, eliminating the need for data to pass through multiple system layers.
- π‘ The internal structure of an FPGA-based trading system includes market data handling, trading logic, order formatting, TCP stack management, and output MAC encoding.
- π§ Designing for FPGAs at Optiver involves avoiding pipelining to reduce latency, focusing on timing closure to save clock cycles, and custom designing IP for the hot path.
- π The use of PCI Express in FPGA trading systems is for status and control, not for the critical trading logic, which remains within the FPGA for speed.
- π Optiver's approach to FPGA design is distinguished by a tight feedback loop, allowing for rapid testing and deployment of changes, with immediate visibility of their impact.
- π‘ The trading environment at Optiver is characterized by a high pace, dynamic changes, and the need for innovative solutions to overcome challenges without introducing latency.
Q & A
What is the primary function of Optiver's trading systems?
-Optiver's trading systems primarily function to improve the market by providing liquidity. They do this by buying and selling various financial instruments such as options, futures, ETFs, and stocks at prices they publish to the market.
How does Optiver's approach to using FPGAs in trading differ from other companies?
-Optiver's approach to using FPGAs in trading is unique in that they focus on minimizing latency. They design their systems to be as fast as possible, avoiding pipeline registers in their critical path to reduce latency and ensure the first-come, first-serve mechanism in trading.
What is the significance of low latency in trading systems?
-Low latency is crucial in trading systems because it allows for faster response times in the market. This is particularly important when multiple parties are competing to execute trades at the same time, as the first order to reach the exchange will be the one that is executed.
How does the shift from traditional trading pits to electronic trading impact the role of latency?
-The shift to electronic trading has made latency a more critical factor. In electronic trading, orders are communicated through network packets, and the speed at which these packets are processed and sent to the exchange can determine the success of a trade.
What are the limitations of a software-based trading system in terms of latency?
-A software-based trading system incurs latency due to the need to traverse the network interface card, PCI Express, CPU, and memory. Each of these components adds a delay, which can be significant in high-speed trading environments.
How does an FPGA-based trading system address the limitations of software-based systems?
-An FPGA-based trading system addresses these limitations by placing the logic needed to make trading decisions as close as possible to the network, effectively reducing the overhead latency associated with PCI Express, CPU, and memory.
What is the role of the FPGA in the architecture of an FPGA-based trading system?
-In an FPGA-based trading system, the FPGA handles tasks such as unpacking Ethernet protocols, filtering network traffic, decoding market data, and formatting orders. This allows for faster processing and decision-making compared to a software-based system.
Why is it important to filter out irrelevant network traffic in an FPGA-based trading system?
-Filtering out irrelevant network traffic is important to ensure that only relevant market data reaches the trading logic. This prevents unnecessary processing of data that does not contribute to trading decisions, thereby saving processing time and reducing latency.
How does Optiver ensure that their trading systems are robust and error-free?
-Optiver ensures robustness and error-free operation by implementing limit checking logic in their trading systems. This helps to prevent issues such as sending too many orders per second or sending orders with prices outside of a controlled range.
What are some of the challenges in designing an FPGA-based trading system?
-Some challenges in designing an FPGA-based trading system include managing the encoding and decoding of market data and orders in market-specific formats, handling TCP/IP stack complexities, and ensuring that the system operates within tight timing constraints to minimize latency.
How does Optiver's approach to design and innovation in FPGA-based trading systems contribute to their competitive advantage?
-Optiver's approach to design and innovation, such as avoiding pipeline registers, investing in the best devices, and hand-designing IP for the hotpath, contributes to their competitive advantage by allowing them to achieve lower latencies and more efficient trading systems.
Outlines
π€ Introduction to FPGAs in Trading
The speaker begins by questioning the audience's familiarity with the relationship between FPGAs (Field-Programmable Gate Arrays) and low-latency trading. He shares his personal experience of being intrigued by an offer from Optiver, a trading firm, to work on an FPGA engineering role. The speaker outlines the agenda for the talk, which includes explaining trading, discussing the architecture of an FPGA-based trading system, and detailing the unique approach Optiver takes in designing FPGAs. Optiver is introduced as a global market-making firm that trades various financial instruments using its own capital and aims to improve market liquidity. The speaker also provides a brief introduction to himself, having been with Optiver for 11 years and having co-written the company's first FPGA-based trading system.
π The Role of Latency in Electronic Trading
This paragraph delves into the significance of latency in electronic trading, especially when multiple sellers compete to trade with a single buyer. The speaker explains how exchanges handle such situations using a first-come, first-serve mechanism, emphasizing the importance of being the first to send an order back to the exchange. The limitations of a software-based trading system are also discussed, including the latency penalties incurred when data traverses through the NIC (Network Interface Card), PCI Express, CPU, and memory. The speaker then introduces the concept of using an FPGA to reduce this overhead latency by placing trading logic closer to the network, providing a more efficient path for trading decisions.
π οΈ FPGA-Based Trading System Architecture
The speaker provides an in-depth look at the internal workings of an FPGA-based trading system, starting with the reception of market data via 10 Gigabit Ethernet. The data is then processed through various stages, including unpacking Ethernet protocols, filtering irrelevant network traffic, and selecting specific instruments of interest. The payload, which contains the core order information, is extracted and passed to the trading logic block, where strategies are implemented. The speaker also discusses the complexities of encoding orders in a market-specific format and the challenges of working with TCP for order transmission, highlighting the importance of maintaining a fast and reliable critical path in the system.
π§ Optiver's Unique FPGA Design Approach
In the final paragraph, the speaker contrasts Optiver's FPGA design approach with that of other companies. He discusses the avoidance of pipelining to reduce latency, the pursuit of challenging timing closures to save clock cycles, and the preference for custom-designed IP over vendor-provided solutions to ensure low latency. The speaker also touches on the environmental aspects of working at Optiver, including the rapid feedback loop, the high-paced and dynamic nature of trading, and the need for innovative solutions to overcome design challenges without introducing latency. The presentation concludes with an invitation for further questions at the Optiver virtual booth.
Mindmap
Keywords
π‘FPGAs
π‘Low Latency Trading
π‘Optiver
π‘Market Liquidity
π‘Trading System Architecture
π‘Latency
π‘Network Interface Card (NIC)
π‘10 Gigabit Ethernet
π‘TCP/IP
π‘Custom Designed IP
π‘High-Paced Environment
Highlights
Introduction to the intersection of FPGAs and low-latency trading, with a personal anecdote about joining Optiver.
Explanation of what trading is and how FPGAs fit into this domain.
Overview of Optiver as a global market-making firm and its role in improving the market.
Introduction of the speaker's background and experience with FPGA-based trading systems.
Historical context of trading from trading pits to electronic trading in data centers.
Importance of latency in trading and how it affects the trading process.
Limitations of software-based trading systems and the latency penalties involved.
Advantages of using FPGAs to reduce latency in trading systems.
Architecture of an FPGA-based trading system and its internal components.
Details on how market data is handled and processed in an FPGA-based system.
The role of filters in selecting relevant market data and ignoring unnecessary information.
Unpacking and understanding the structure of market data for trading decisions.
Implementation of trading logic within the FPGA and strategies for order execution.
Challenges in encoding orders in market-specific formats for exchange acceptance.
Importance of limit checking in trading systems to avoid costly mistakes.
TCP stack considerations and the challenges of state management in trading systems.
Optiver's unique approach to FPGA design, emphasizing speed and custom solutions.
Environmental factors that differentiate Optiver's approach to FPGA design and trading.
Innovative solutions and out-of-the-box thinking required for latency reduction in trading systems.
Invitation to the Optiver virtual booth for further questions and discussion.
Transcripts
[Music]
right well good evening
um let me start with a question uh who
have you out there have thought about
the
intersection between fpgas and low
latency trading
um i know i hadn't uh thought about it
much
i got a phone call about 11 years ago
from optiver
asking me if i was interested in an fpga
engineering role
and i remember thinking to myself what
on earth is this company doing
with with fpga they're a trading firm
well
in the next 15 minutes i hope to give
you a little bit of insight into
why and how we use fpgas in this domain
a little agenda for this talk first i'll
explain a little bit about what
trading really is and that can give me a
basis for explaining how fpgas actually
fit in
to this domain um i'll then talk a
little bit about
the architecture of an fpga-based
trading system
and finally i'll discuss a little bit
about how the
how we approach design for fpgas at
optiver and how it's a bit different
than
how other companies which makes use of
fpgas approach them
a little introduction maybe first on
optifer and myself
optiver itself was founded in 1986 in
amsterdam
it's a global market-making firm it
means we're a training firm
we trade all kinds of things but
predominantly options
futures etfs stocks and that kind of
thing
and we do all of this with our own
capital we don't have any clients
this is all done internally
and what's our goal here in the
marketplace it's to improve
the market and we do that by showing
liquidity and what does that mean
that means if if you want to buy
something we'll sell it to you
if you want to sell something we'll buy
it from you but always at prices that we
are
publishing uh yeah to the to the market
at large
um we are a global company
so we've got offices around the world
and from those offices we can cover
exchanges around the world um amsterdam
of course which is well where i'm based
but we also have offices in london
chicago sydney and shanghai about me
personally i've been with optiver for
about 11 years now which maybe tells you
something about how my phone call with
them went 11 years ago
um and in that time i've i co-wrote
the first fpga-based trading system that
we had
and went on to design and implement and
uh
break uh numerous other systems in the
last
11 years to bring to one add new
features and two to to bring
these systems to new exchanges
so what's trading really um yeah we see
we have a photo here this is a trading
pit this is the way that trading was
done
uh well and still is done to some degree
for uh 400 years
it's been done like this people yelling
at each other and
a cluster um and what's actually
happening here is that
one of these people will be saying
something like i want to buy
five shares of of apple for 10 euros
and anyone around this person or at
least with an earshot
uh can can hear that and think oh is
that a good deal or not
and if it is then they can yell back i
want to sell five shares to you
and that's how a trade is done now
in the last 25 years um this type of
trading has largely been replaced with
electronic trading
and that what does that look like it
looks like looks like this
uh these days so this is a data center
and
what happens now is um an exchange
a market can rent physical space um
in in this data center where they can
install their own servers their own
network equipment and this kind of thing
but so can market participants can also
um
rent space uh in in this in this data
center and install
their own equipment but how do we
actually
communicate here and that's that's one
of the keys and
and the old style it was by yelling in
this new style electronic trading it's
by
sending ethernet packets so these data
centers will provide
10 gigabit network connectivity
generally
which you can use to communicate with
the exchange
so now when somebody wants to buy five
shares of apple for
10 euros they have to encode that
information into an
into a network packet and send it to the
exchange's
central server at that point the
exchange can
rebroadcast that information to all
market participants now
where does the latency angle come in
where did yeah where did the lazy come
in here
so imagine that more than one person
wants to sell to the single buyer
imagine a thousand people
want to sell to the single buyer yeah
how do we solve that problem because
from the exchanges point of view there's
going to be a thousand
orders all coming in at the same time
here at the same time
um and one of the ways that an exchange
can solve that problem
is by implementing a kind of first come
first serve
style mechanism and then you can start
to see
why the latency angle starts to matter
which is yeah the first one to get their
order
back to the exchange will be the one who
can affect the trade
um so knowing that latency is
important um what are the limitations
of a software-based trading system um
here we can see uh on the on the left
we have our exchange who's going to be
broadcasting information
and then it's going to hit our server
and on the server we're going to have a
software application which
you know needs to understand what's
happening in order to take a trading
decision
but in order for the software
application to see this we're going to
have to first traverse
the nic so this is a network interface
card we're then going to traverse
pci express and then we get into our you
know our cpu and memory
um where we're going to have you know
our application running
now in this chain um we're gonna be
incurring a latency penalty
crossing this guy crossing pci express
but of course even software applications
are gonna have their own
variance and latency with respect to
getting access to an
actual cpu core imagine the colonel
wants to start doing something on the
processor you're running on and you get
booted off
that's going to incur a latency hit as
well on the memory side
imagine that the memory you're trying to
access gets dropped out of cash
and you're gonna have to go you know
further away to get the information you
actually need
it's also gonna introduce a lot more
variance in your in your latency
because ultimately what we're trying to
do here is get market data
into your application take a trading
decision and get back out again
and there's a lot to accomplish there so
how can fpgas
help us in this next slide
it doesn't look that much different but
there is a key difference which is
the nick has been replaced with an fpga
so what if we can cut away all of the
extra overhead latency of of pci express
and our software application and cpu and
memory
problems and we can do that by trying to
place
all the logic we need to take a trading
decision
as close as possible to the network and
that would be here in the fpga
so if we if we dive deeper into the fpga
we can actually start to see
what what the skeleton of a of an fpga
based trading system looks like
internally um so what do we have here on
the
on the top left so we'll have our market
data so this will be
10 gig ethernet um some you know
network traffic being broadcast
indicating the state of the market
somebody wants to buy somebody wants to
sell that kind of thing
what do we have to do in order to take a
trading decision on this information
well one of the first things that we're
going to need to do is kind of unpack
the lowest level ethernet protocols here
there's some physical layer things we
have to take into account
and we use a mac to help us with this
but the output of this
uh here will be effectively a a packet
stream
a network stream
[Music]
because this is ethernet there's going
to be a number of
networking headers on top of the actual
payload that we
care about so there's going to be
ethernet header an ip
v4 header a udp header and these kinds
of things
and these what we can implement here in
this first block
in the market data handling block is a
filter
um on these networks there's going to be
other networking equipment
blabbing stuff we don't really care
about maybe like routing protocol
updates and that kind of thing and we
don't want any of that to hit our
trading logic that's stuff we want to
ignore so we can filter out
based off of you know higher level
header information
secondly we can use these filters to
only select the instruments that we
really
care about generally exchanges will
partition the
the the stocks uh that that are being
traded on these exchange by multicast
group
so if you know you only care about a
subset of instruments you can also apply
a filter
at this stage to only you know get
access to the instruments you really
care about
so once you've gotten rid of your your
networking headers uh you're left with
just your payload
and this is where you know the meat
of the um of the order message really is
that
somebody wants to buy five shares at 10
euros
now every exchange tends to encode all
this information
and their own native format so you'll
have to consult you know the relevant
exchange
but generally what the way this will
work is that uh the first couple bytes
of the payload will have some kind of
information which tells you
what the structure of the rest of the
packet looks like
so that could be that this is a trade or
that this is the
end of the trading day and it's time to
turn off or it could be
that somebody wants to buy five shares
of apple for 10 euros
so once you once you know what kind of
message this is
then you're gonna start to unpack the
stuff behind it
and that could be like the next four
bytes could indicate
uh the instrument id so is it is this
apple is this google is it tesla is it
whatever um and then you behind next
four bytes behind that could be
this is the price that somebody's
willing to pay represented in
as a as a single precision float for
example
and then the following four bytes could
be the volume how many shares do someone
want to buy
so once you've unpacked all that
information in the market data handling
block
you can then pass it downstream to your
to your trading logic
i won't go into much detail about what
we do at optiver here
um but you could implement you know your
very simple strategy which could just be
if anyone is ever willing to buy apple
for more than
eight euros then sell sell your share of
apple to that person
so yeah up to you to implement what you
actually want in this blog
um leaving this block you'll have your
order you need to send something back to
the exchange to cause a trade to happen
um and what one of the problems that
you'll face
is that just like the market data is
encoded in a market specific
format so are the orders so we need to
make sure that all the information is
placed in the right
spot and encoded in the right way and
that can also be quite a challenge
because sometimes exchanges
like really um unfortunate encodings
like ascii sometimes we need to convert
like binary numbers to ascii
representation
to make them accepted by the exchange
which is not that fun
um so that's that's what kind of logic
you would actually need to
uh build inside your order formatting
block
say additionally another critical thing
to maybe put in this block would be some
kind of limit checking
logic so because we designed these
systems to be as fast as possible
um any any mistakes or any bugs or any
you know um behavior you don't expect
can be very costly so it would be a good
idea to put a limit checking
uh code in here to make sure you don't
send for example
too many orders per second or
that's the prices that you're sending to
the exchange fallen
within some bound uh that you that you
control
uh once you've gone through all that
then we're gonna get to our tcp
uh stack and yeah well one of the
one of the um things that i think almost
every exchange expects
is that the orders that you place going
back to the exchange are encoded in
tcp um which is a bit of a headache
for us doing tcp means a lot of state
tracking there's sequence numbers to
worry about that kind of thing
and as well there are features of tcp in
fact the hallmark of tcp is that it's
reliable
um which means you know your your
message your segment always gets the
other side
but when you really zoom in on what that
means it means that if you send
a segment and it's never acknowledged by
the remote
that you need to send it again and that
means a lot of
state management buffering and that kind
of thing um which is a bit of a headache
finally uh we'll get to the to the
output mac here where we can again
re-encode this information
at the lowest level to be re-transmitted
back to the exchange
to try and make our our trade happen
now we call this this loop we call this
our our critical path
or a hub path right i mean this is we
try to make this path as fast as
possible
um as of a couple of years ago uh we
were
well below 200 nanoseconds um
that would be from from the time that
the you know the first bit
hits the pin of the network connector uh
on the fpga card
until the first bit hits the pin on the
on the egress side that went back to the
exchange
um one one element i didn't touch on on
this is pci express
we still do make use of this um and it
we
make use of this for um status and
control
so we will have a software application
still running on a traditional
server above us um but that's used
for a couple of reasons one is yeah
making sure that whatever logic
you're doing here and the kind of orders
that you're sending back to the exchange
look sane and sensible and make sense
but as well control so when when does it
make sense to turn this thing off
um do we want to do more configuration
on
what kind of multicast groups do we let
through
that kind of thing
uh so finally last slide uh how do we
approach
design differently um at optiver
compared to other companies which make
use of fpgas
first one is the technical side um
there's a couple things here one is
pipelining
um yeah i think for most you know
digital designers
you know they're taught to pipeline
their design right that's just how you
how you do it you put a register on this
on the input and register on the output
and that's just what you do
for us that's that's almost never what
we want to do um
because every register in our hot path
is
latency and we just can't afford to do
that
um an interesting flip side of this
point is that for time and closure um
you know i think for most engineers when
when time enclosure is easy that's great
that means you don't need to worry about
it and you can go on to think about the
next thing
um for us that's not really the case if
time enclosure is too easy
then that probably means there's a clock
cycle somewhere
we can save so we'll deliberately try
and make timing harder
uh well as a result that we can save a
clock somewhere uh yeah we can
we can make it faster so the net result
is that
kind of perpetually in a state where
timing is irritatingly hard to meet
but not impossible um which i don't
think is very normal elsewhere in the
world
um second point here is as best devices
yeah
you know we're we're really on the on
the cutting edge here we will
invest in buying whatever we need to do
to make sure that we are on the on the
on the best possible footing we can be
um
the last point here is around custom
designed ip a lot of
fpga vendors of course provide a lot of
um
you know pre-built pre-packaged ip for
you to use like fifos and that kind of
thing
we don't generally make use of this
stuff at least not on our hotpath
uh most most of this um ip it's robust
and tested
but it's not designed for low latency
purposes so we tend to hand design
everything in our hotpath
uh the second class of things on how we
approach things differently
is more environmental one is
we have a real tight feedback loop on
our work um
i think whereas in some companies you
might not see the result of your work
for
for weeks or months or maybe even years
in our environment you could be making a
change to the code base
you know test benching it going through
a regression suite
but seeing you know a release out the
next day and having it deployed and
running
and you can see the results of your work
literally the next day
um not only seeing it running but also
seeing its impact
on let's say the success of a trading
strategy
we are so dependent on on latency for
some of these things that you can really
see the impact
um directly the second point
is around being a high-paced environment
trading is really dynamic um
there's things are changing all the time
around the world exchanges change the
way that they
operate um opportunities we perceive uh
um come quickly so there's
very rarely a dull day i think in
october there's always something new
uh coming i think the last point
um i wanted to highlight was on yeah
innovative solutions
um yeah tying into the fact that we
don't pipeline for example
means that in order to solve some of the
problems that we have
without introducing latency you have to
get real out-of-the-box thinking
and that could be thinking about novel
ways to
achieve a comparison or to accomplish
some
logic um which yeah it's nice to always
have that kind of fresh challenge
uh present on on a daily basis
um so yeah i think that was that was my
presentation
um thanks for joining me for this
i don't think i have much time to take
questions now
however i will be present in the in the
optiver
uh virtual booth i believe at 7 15
european time so happy to take any more
questions or any
any other things you want to know happy
to talk in that venue
[Music]
thanks
Browse More Related Video
Open High Low Close - Understanding Candlesticks
Trading with $10 to Prove it's Not Luck...
Perfecting LTF Orderblock Entries With CRT - Candle Range Theory - ICT Concepts
Top Swing Trade Strategy with Exact Entry and Exits
Trading Cepat dan Mudah dengan Market Structure (Detail)
The Strategy I Wish I Knew as a Beginner
5.0 / 5 (0 votes)