Precision Time Protocol Profile for Data Center Applications & Related Network Requirements

Summary

The talk introduces the PTP (Precision Time Protocol) profile for data centers, aimed at improving time synchronization services. Michelle and Thomas from OCP TAB discuss the profile's development, its objectives, and its applications in enhancing distributed databases, network monitoring, and 5G synchronization. They also cover hardware advancements in time synchronization and the profile's technical specifications.

Takeaways

• 😀 The presentation discusses the PTP (Precision Time Protocol) profile, which is a set of standards and options tailored for data centers.
• 🔍 Michelle from Mera and Thomas from Nvidia led the effort in developing the PTP profile for data centers, working with many others in the community.
• 📈 The PTP profile aims to define how to integrate various time synchronization technologies to improve data center applications.
• 🕒 The core objective of the PTP profile is to enhance time synchronization service in data centers, moving from millisecond precision to microsecond precision with high reliability.
• 📚 The PTP profile document, developed within the OCP TAB, explains various options and requirements for implementing PTP in data centers, including network topology, time error performance, and clock types.
• 🔄 The profile addresses applications like distributed database systems, network monitoring, and 5G synchronization, aiming to increase transaction throughput and provide reliable air interface synchronization.
• 🌐 The PTP profile specifies the use of hardware timestamping to achieve sub-10 nanosecond accuracy and resolution, moving away from the earlier software timestamping approach.
• 🔄 The profile includes a reference model that decomposes the problem into three layers: time reference, network fabric, and server layers, each with specific roles in time synchronization.
• 🔄 The profile defines a time error requirement of plus or minus five microseconds between any two servers within a data center, ensuring high precision in time synchronization.
• 🔄 The PTP profile for data centers initially uses a model with only transparent clocks, but future work will explore a model with boundary clocks for more flexibility.

Q & A

What is the primary purpose of the PTP profile?

The primary purpose of the PTP (Precision Time Protocol) profile is to define a set of standards and options that can be tailored to meet the needs of data centers, improving time synchronization services and enabling new applications.

Who were the key contributors to the PTP profile for data centers?

Michelle from Mera and Thomas from Nvidia were the leading figures in the development of the PTP profile for data centers, along with contributions from other people and companies.

What is the main objective of the OCP TAB in relation to time synchronization?

The main objective of the OCP TAB (Open Compute Project Technical Advisory Board) in relation to time synchronization is to define a high-level time synchronization service across data center infrastructure, aiming to improve current applications or enable new ones.

What performance improvement is targeted by the PTP profile in data centers?

The PTP profile aims to provide two to three orders of magnitude better performance in time synchronization compared to current network timing protocols used in data centers, moving from milliseconds to microseconds precision with high reliability.

What are some applications that have been discussed within the PTP project group?

Applications discussed within the PTP project group include distributed database systems, network monitoring, and 5G synchronization. These applications aim to increase transaction throughput, measure network events more precisely, and provide reliable air interface synchronization.

How does the PTP profile help in maintaining the order of transactions in distributed systems?

The PTP profile helps in maintaining the order of transactions by ensuring that any committed timestamp is always in the past relative to a reference block, minimizing clock skew and thus improving the performance of distributed systems.

What is the significance of hardware timestamping in PTP implementations?

Hardware timestamping is significant in PTP implementations as it allows for more accurate time stamping closer to the hardware layer, reducing the impact of system noise, latency, and other factors associated with software timestamping. This leads to better accuracy and resolution in time synchronization.

What are the two models planned for the data center profile in PTP?

The two models planned for the data center profile in PTP are Model 1, which uses only transparent clocks and relies on network routing for failure recovery, and Model 2, which uses boundary clocks with each switch device running a boundary clock for processing messages hop by hop.

What is the time error requirement defined in the PTP profile for data centers?

The time error requirement defined in the PTP profile for data centers is that the difference between any two servers' PTP clocks within a data center should be within plus or minus five microseconds.

What is the call to action for the PTP profile development in data centers?

The call to action is to invite people to join the work stream number two to develop the second version of the profile, focusing on the use of boundary clocks, security aspects such as authentication and verification of PTP messages, and load balancing of the PTP unicast sessions.

Outlines

00:00

🗓️ Introduction to PTP Profile for Data Centers

The speaker introduces the Precision Time Protocol (PTP) profile, which is a set of standards and options tailored for data centers. Initially, there was no specific profile for data centers, but Michelle from Mera and Thomas from Nvidia led the effort to create one. The PTP profile document outlines various options for time synchronization and explains what should be avoided. The presentation aims to provide a walk-through of this profile, highlighting its importance in defining time synchronization services for data centers.

05:01

🔍 Applications and Objectives of PTP Profile

This paragraph discusses the applications and objectives of the PTP profile in data centers. The main goal is to improve time synchronization services to enhance current applications or enable new ones. Examples include distributed database systems, network monitoring, and 5G synchronization. The PTP profile aims to provide two to three orders of magnitude better performance than current network timing protocols, moving from milliseconds to microseconds precision with high reliability. The speaker also mentions the various work streams and the PTP profile specification.

10:01

🌐 PTP Profile Development and Industry Adoption

The speaker highlights the advancements in time synchronization distribution and how PTP has been adopted across multiple industries. Each industry has developed a PTP profile specification that defines the capabilities required for their specific use cases. However, the data center industry was previously missing a PTP profile, which has now been developed within the OCP TAB. The new data center PTP profile is a comprehensive document covering network topology, time error performance, clock types, communication modes, and more. It was released in September and is available for review.

15:02

📈 Time Error Requirements and Reference Model

This paragraph focuses on the time error requirements and the reference model for the PTP profile. The goal is to ensure that the difference between any two servers' PTP clocks within a data center is within plus or minus five microseconds. The speaker explains that each PTP clock in the servers must be within plus or minus 2.5 microseconds of a common reference, such as GPS or GNSS. The reference model is also discussed, which includes the time reference layer, network fabric layer, and server layer, detailing how time is recovered and passed on to applications.

20:03

🛠️ Hardware and Software Timestamping in PTP

The speaker discusses the evolution of hardware and software timestamping in PTP. Initially, hardware was not capable of drawing timestamps, leading to software timestamping, which had drawbacks due to system noise and latency. Modern implementations now use hardware timestamping, which provides sub-10 nanosecond accuracy and resolution. The speaker also covers the transition from two-step to one-step clock mechanisms, explaining how hardware timestamping allows for more accurate and reliable time synchronization.

🔄 PTP Profile Models and Future Developments

The final paragraph covers the current PTP profile models and future developments. Model 1, which uses only transparent clocks, is currently in use, allowing for accurate time synchronization through network routing. The next phase involves exploring Model 2, which uses boundary clocks with each switch device running a boundary clock. The speaker also mentions the need for failure mechanisms and the upcoming work on security aspects, authentication, and load balancing of PTP unicast sessions. A call to action is made for participation in the development of the second version of the profile.

Keywords

💡PTP Profile

The PTP (Precision Time Protocol) Profile refers to a set of standards and options tailored for specific applications or environments. In the context of this video, it is a document that explains how to configure PTP for data centers. The PTP Profile is crucial as it defines the options and standards to achieve high-precision time synchronization, which is essential for various applications in data centers.

💡Data Centers

Data centers are facilities that house computer systems and associated components, such as servers, storage devices, and networking equipment. In this video, data centers are the focus as the PTP Profile is being discussed in relation to their needs. The goal is to improve time synchronization within data centers to enhance performance and enable new applications.

💡Time Synchronization

Time synchronization is the process of ensuring that all devices in a network have the same time. This is critical in data centers for coordinating operations and ensuring the reliability of applications. The video discusses how the PTP Profile aims to improve time synchronization to the microsecond level, which is a significant improvement over current standards.

💡OCP TAB

OCP TAB (Open Compute Project Technical Advisory Board) is a group within the Open Compute Project that focuses on developing and maintaining standards for data center hardware and software. In the video, Michelle and Thomas from OCP TAB are leading the effort to develop the PTP Profile for data centers, indicating the importance of this standard in the industry.

💡Distributed Database Systems

Distributed database systems are databases that are spread across multiple sites, servers, or locations. The video mentions that one of the applications where improved time synchronization is beneficial is in distributed database systems. Better time synchronization can increase the throughput of transactions by ensuring that all parts of the system are in sync.

💡Network Monitoring

Network monitoring involves measuring and analyzing network traffic and performance. The video discusses how precise one-way delay measurements, enabled by better time synchronization, can improve network monitoring. This can provide better visibility into network events and help in troubleshooting and performance optimization.

💡5G

5G is the fifth generation of mobile networks, promising faster speeds and lower latency. In the context of this video, 5G is mentioned as an application where reliable air interface synchronization is crucial. The PTP Profile can play a role in ensuring that 5G networks maintain precise timing, which is essential for their operation.

💡Hardware Timestamping

Hardware timestamping refers to the process of capturing time stamps at the hardware level, rather than in software. This method is more accurate and less susceptible to system noise and latency. The video explains that modern implementations of PTP use hardware timestamping to achieve sub-10 nanosecond accuracy and resolution.

💡One-Step Clock

A one-step clock is a mechanism in PTP where the timestamp is directly written into the message itself, rather than using a two-step process. The video discusses the advantages of one-step clocks, such as avoiding out-of-order sequences and ensuring that each message is linked to its own timestamp, which is crucial for maintaining accurate time synchronization.

💡Boundary Clock

A boundary clock is a type of clock used in PTP networks that can act as both a master and a slave. The video mentions a future model for the data center profile that involves using boundary clocks, where each switch runs a boundary clock and processes messages hop by hop. This approach allows for more flexibility and can improve the reliability of time synchronization.

💡Best Master Clock Algorithm (BMCA)

The Best Master Clock Algorithm is used in PTP networks to determine the best source of time. In the context of the video, BMCA is mentioned in relation to the use of boundary clocks, where it helps decide which path to use for time synchronization. This algorithm is crucial for maintaining accurate and reliable time distribution in complex networks.

Highlights

Introduction to the PTP profile for data centers, highlighting its importance in tailoring standards and options for time synchronization.

Michelle and Thomas from OCP TAB leading the effort in defining the PTP profile for data centers.

The PTP profile document explains various options for time synchronization and what should be avoided.

Objective of the PTP profile is to define how to put together time synchronization components for data centers.

OCP TAB's focus on defining a high-level time synchronization service to improve data center applications.

Precision Timing Protocol (PTP) is chosen for its potential to provide two to three orders of magnitude better performance than current network timing protocols.

Discussion on applications like distributed database systems, network monitoring, and 5G synchronization as use cases for PTP.

Example given to illustrate the importance of accurate timestamps in transaction ordering and real-time operations.

Advancements in time and synchronization distribution, including silicon support for PTP and PTP-aware switches and routers.

Different industries have developed PTP profile specifications tailored to their specific needs.

Data centers have been missing a PTP profile, which OCP TAB has addressed by developing a PTP profile for data center applications.

Reference model for PTP in data centers includes time reference layer, network fabric layer, and server layer.

Time error requirement defined as plus or minus five microseconds for any two servers within a data center.

Hardware timestamping versus software timestamping discussed, with hardware providing sub-10 nanoseconds accuracy and resolution.

One-step versus two-step clock mechanisms explained, with modern implementations favoring one-step for accuracy.

Introduction of model one using only transparent clocks for the data center profile, and the upcoming model two with boundary clocks.

Failure mechanisms using IGP and best master clock algorithm in the data center profile.

Call to action for participation in the development of the second version of the profile, focusing on boundary clocks, security, and load balancing.

Transcripts

ready

okay so uh next talk is gonna be remote

so i will just here like stand here to

change the slides

uh it's about the ptp profile which

basically

when it comes to ptp and profile is

basically a set of standards set of

options that you can take

and when we started

tap there was no

profile pdb profile like tailored for

data centers

so uh

michelle from mera and uh thomas from

nvidia they worked plus a lot of

other people here but

you see these two names like uh they

were leading the effort

on uh various uh

let's say a document the outcome of this

was a document that basically explains

various options that you can take and

tailor it to your needs and also

explains like

what you shouldn't do perhaps so with no

further ado let's

get it started so do we have

michelle or thomas online

yes i am yeah hey michelle thank you

okay so let's do it michelle

all right uh

very good um if we can go to the slide

please yeah we are at the agenda slide

okay super okay thank you yeah uh thanks

a man yeah so um

my name is uh michelle from uh i'm here

with my colleague thomas to

basically give you a quick short walk

through of the

first ptp profile

for the data center

industry or community that you know

we've been working on within oct tab in

the past year we've made a lot of great

progress

um and this is what we want to share you

know with all of you

um a lot of things were mentioned in the

previous stocks for instance i heard you

know

gnss

open time server time card unicast

reliability and all that

and the ptp profile like ahmad was

saying in the introduction

is really a no more than a document that

basically defines how you put all of

this together

okay so this is really uh you know the

core of the objective the ptp profile is

to define

you know how we put all of that together

uh could we uh go to the next slide

please i'm on

so

um when you look at the work you know

that is being accomplished within the

ocd tab one of the main objectives

is to define you know very high level

this aspect of a time synchronization

service

across you know the infrastructure of

the data center to basically

improve either a set of current

applications or enable

you know a new set of applications and

we'll hear you know about some of these

applications a little bit uh you know

later this morning

um

right now uh you know within the ocd tab

we've converged on using the precision

timing protocol

you know with with a high level

objective to provide two to three orders

of magnitude better performance

you know when you compare to you know

current network timing protocol

infrastructure that is used in uh data

centers uh

you know uh today

essentially we want to move from you

know milliseconds

right which is still a you know a small

number in terms of time

down to the micro session microseconds

you know precision with a fairly high

level

you know amount of uh reliability

and you know to to realize that um we've

been working on multiple work streams

that were uh presented the

uh previously

um and one of them relates to the ptp

profile specification and this is what

we'll introduce uh this morning myself

and thomas

we cannot see the slides

yeah thank you all right

so

in slide three

several applications you know in the

past year have been discussed within uh

you know uh

the tapa project group

through various presentations you know

given by community members you'll find a

link there with you know all of the

recorded

talks

um very high level for instance

there were a couple of talks on what we

call distributed database systems where

the objective there is to increase the

throughput you know of transactions via

the use of better clocks

the second type of application that was

discussed is relates to network

monitoring which basically the objective

there is to

try and you know measure network events

using what we call one-way delay

measurements so

if you make your one-way delay

measurements more precise

then you know in theory that should

you know give you better visibility to

what is happening you know with the

events you know in the network

uh there were a few talks also on 5g

where the objective there um and this is

a well-known you know use case to

provide reliable

you know air air interface uh

synchronization and we'll hear a lot

more on this subject in the telco uh

stream this afternoon there are a couple

of talks on that specific uh subjects

um next slide please

so let's take a a very high level

example for instance to put you know

this into uh frankincense context

let's say you've got an application a

that basically issues a write command

you know to a client here called c1

basically that client when it receives

that request it chooses a time stamp

let's say t1 that timestamp might be in

the future and then it executes you know

that rights to a set of replicas

when all of this is completed and you

know the right operation has been

acknowledged

the same application might for instance

design you know to do a read but it does

so in this example for instance through

a different client

uh that is called the c2

so again c2 chooses a read time stamp

that time stamp might be for instance in

the future also that time step is you

know time t2

and it basically reads you know the

object from you know one of the replicas

in this example here just show uh

through r3

so here's basically you know a bit the

dilemma

if this timestamp t2 is greater than t1

then the client c2 is going to be

reading valid data right

i mean it might take a bit of time to

read the data

right between the difference between

where t2 is in comparison to t1

but if it's in the future it's going to

read you know the proper uh valid data

but if t2 for instance is smaller than

t1

because of things like you know a clock

skew

the application will basically see

um stale you know data

even though the right you know what that

was done by client c1 completed before

you know uh

the the read operation began

you know when you look at things you

know from an ordering or real-time

ordering of operations uh or

transactions

so essentially what you want to make

sure here in this example is that any

committed timestamp

is always in the past relative to a

reference block

and in some of those database systems

maintaining the ordering of these

transactions is very important but also

making sure that you have very you know

the smallest clock skew possible

are basically drivers to increase you

know the performance of these type of

systems

next slide please alan

yeah thank you yeah so

um there's been you know a lot of a

significant advancement uh in the

distribution of a you know a time and

synchronization

in the past you know certainly in the

past decade

primarily around things like we've heard

this morning you know a silicon that

supports pvp

switches and routers also that supports

you know that are ptp aware oscillators

you know a linux ptp stack for instance

test equipment all of which you know uh

support uh

pdp and because of that multiple

industries i've adopted ptp as you know

the protocol or the technology of a

choice from any use case scenarios

um

and there's many you know uh ptp

networks in operation

in each of these

industries and we'll go into some

details in the next slide each of these

industries has developed what we call a

ptp profile specification

which essentially defines the

capabilities required to support a use

case

you know scenario for their particular

uh

industry so the profile really provides

you information on how you implement

things how you configure and how you

will basically operate the ptp

next slide please

so this is basically a you know a table

where today there is

about half a dozen ptp profiles that

exist in the industry today each having

a different you know scenario telecom

mobile professional uh you know a video

power and so on

um

but one in the in the one industry

that you know has been missing out

you know from uh you know uh from this

was the data center and this is what we

did within ocp tab within you know the

past year is to basically develop

a dtp profile for the purpose of

data center applications

um

the

and this is the last row in that table

the dtp profile is basically a 20

you know plus page

document it was contributed by um

six different uh companies and it re

it

basically goes through

and contains various requirements that

for instance pertain to things like

network topology

uh what is the expected time error

performance

what are the type of clocks for instance

and we've heard that a little bit

earlier from japan this morning right

whether it's transparent clocks versus

boundary clocks what are the

communication mode uh the pdb messages

uh the message rates unicast

communication and so on all of that is

defined in that document and that

document was released back in september

and is available on the uh

ocp uh contributions uh web page uh for

anyone to go and

read and digest yeah

uh next slide please

so one of the things we needed to in

order to kick off this activity we

needed to come up with a reference model

so very high level we decompose the the

problem statement into three layers

what we call the time reference layer

which contains you know your gnss your

gps

your rooftop antennas your

open time server your time cards um and

then the second layer is what we call

the network fabric layer which is

basically a large set of switches you

know or routers for instance that are uh

you know ppp aware uh for example in the

first profile these are uh transparent

clock

capable switches

and then the bottom layer is what we

call the the server layer or where you

have a very large ship of servers that

are also pdp aware through what we call

a clock that is called the ordinary

clock

this is the clock that its

responsibility is to recover time and

then pass it on you know to the

application that's where you know the

demarcation between the ptp network and

the application

uh

resides

next slide please

my last

slide before i pass it on to thomas

a man next slide please

okay thank you

yeah

um the second thing that we did

and this is a you know quite important

here it was to define the time error

requirement okay what are we trying to

meet here what is the expected you know

our performance

if you look in the right hand you know

bottom corner of that slide you're going

to see this number five microsecond

this is what we basically

came up with as a requirement

essentially that says that if you would

pick any servers

within a data center and you could

measure

for instance their ptp clock

that the difference between any two of

these servers

right would be within plus or minus five

microseconds okay this is the absolute

value here five microseconds within plus

or minus five microseconds so one way to

implement that requirement

is to say

that

um

each ptp clock that exists

into for instance the servers

has to be within plus or minus 2.5

microseconds

of a common reference that common

reference being that reference that is

at the top of that tree topology here

for instance gps or gnss as an example

so if you take an example for instance

you take two machines

or two servers

you know one server is minus 2.5 and the

other one is plus 2.5

they have a difference in between the

two of five microseconds

and vice versa right um any combination

there of

of you know these values as long as it's

within

2.5 microsecond of a a common reference

will basically satisfy that condition of

uh

five micro seconds between any two uh

servers

so um

the um the ptp profile addresses you

know how you put all of that together it

talks a lot about you know some of those

you know requirements talks a lot about

you know uh

a lot of these uh building blocks uh

here and you know i i encourage you know

all of you to go and download that

document and you know uh read it uh

through

with that i'm gonna pass it to uh thomas

we'll provide

more information on some of the details

of what's in the ptb profile quick

reminder you have less than five minutes

left

okay thank you michelle

so i'm going to try and

and speed up time

and try and make sure we reverse the

clocks since we've only got a bit of

time left

so how do we actually achieve uh the

target requirements that uh that was

mentioned on the previous slide well

first of all we're going to go through a

bit of

what goes on in the hardware of switches

in order to provide accurate time

stamping

so historically uh when the first

generations of the

ptp standard came out back in 2002

most of the hardware wasn't capable of

drawing

timestamps so it was a software model so

software

timestamping was the way forward at that

point in time but that has a number of

drawbacks because in the case of

software timestamping everything is done

well obviously in software which means

it will

be

impacted by

system noise from the operating system

latency through the pipeline scheduling

and everything else so as you can see

those ptp message were carried from the

interface up through the fi the mac the

a6 to the operating system where the ptp

stack itself would reside so the ptp

stack the so-called virtual disciplined

clock and the time stamping all occurred

in user space

with hardware timestamping

we actually went into the opposite

direction which is what we want we want

to be able to timestamp as close as

possible to

the hardware layer i defy

so modern implementations will actually

do the timestamping itself in the phi

and that's also where we reside the uh

phc the ptp hardware clock

the stack itself of course is still

running in user space

but that actually means we are able to

have modern implementations that are

able to do sub 10 10 nanoseconds uh

accuracy and resolution in those

environments

next slide please

yeah so the other side of the uh

the story is beyond the hardware versus

software timestamp is we have one uh or

one step versus two-step clock and again

for

historical reasons the original

implementations

were not capable of

not only drawing hardware timestamps but

even

due to that we were using a two-step

mechanism so since we're doing software

implementations we were using a two-step

software environment which meant that

when you actually draw a sync message

from your uh from your source uh you

were not able to have the accuracy to

actually write the uh

the timestamp in the message itself so

you would send what we call a follow-up

message

now

that was also the case because uh the

message rates or i should say actually

the interface rates um the timing is

available to encode that timestamp into

the

into the interface at that speed uh was

not uh was not compatible with the

implementations these have gone away i

mean in 2022 we can do hardware

timestamping uh directly at uh you know

100 gigs and beyond so this is uh this

problem is moved away the other thing

also with one step is that guarantees

that each message is linked to its own

timestamp what you want to avoid is that

the sync message would take one path and

the follow-up would take another path

and that could happen if you've got you

know equal

cost multi-path across multiple links

and you actually pack it spraying those

messages which means you could have out

of order sequences

next slide

yeah

so where am i

uh no i think there was a slide in

between no

yeah thank you

so um having said that about uh the

one versus the two-step clock

we also have two models that are planned

for the data center profile right now

the model that michelle was talking

about is model one which exists and uses

only transparent clocks so as we've

mentioned uh this means that we can by

using the one step hardware clock we can

avoid uh spraying and we're actually

using network routing for the

failure recovery so whatever path there

is between

the oc in the grand master through the

chain of switches

your igp will actually be sorting out

making sure those messages arrive

between one end and the other

in the next phase of the development of

the data center profile we're looking at

model 2 which is currently a suggestion

using a boundary clock and in this case

every single

switch device runs a boundary clock and

that boundary clock will ultimately be

processing messages hot by hop

since it'll be disciplining its own

phc and it will become the

source for the device that is downstream

what we're doing is we're actually

comparing the data set that is provided

from the pdp announce messages and the

failure mechanisms is using the bnca the

best master clock algorithm to decide

which path is being used between those

devices

again

in earlier implementations of boundary

clocks there were issues related to

settling time and

oscillation

that has been greatly reduced in modern

implementations but it is

not as 100 clean cut as it is with a

transparent clock

and this model you can use one or two

step also which gives you flexibility

and this is our next work item for the

data center profile

uh thomas uh we have uh less than a

minute left so yeah

i'm i'm getting as fast as i can yeah so

failure mechanism

um it's the same model as was talked

about before we've got the igp

we're carrying it

those messages across the transparent

clock and the endnote itself will figure

out from the data set

which grand master he wants to use so

it's uh it's pretty straightforward and

it's dictated by the data set

the next slide

and nearly at the end

we have the data center profile which

gives you a recap which you can read on

screen

of the different attributes are provided

we know that all devices need to be time

aware we know we can run over

different transports we've got default

message rates to define this is what you

have in profiles traditionally

we're using v6 uh unicast as the main

mechanism here

and we have different levels of accuracy

and clock classes they're defined so

we'll go to the last slide

which is the call to action

in the 10 seconds i've got less

so

we want or we're looking for people to

help us out with the work stream number

two to develop the second version of

profile dimension the boundary clock

we're looking at security aspects how do

we want to

add authentication and verification of

those ptp messages and the load

balancing of the pdp unicast sessions so

please come and join us in this effort

and

we're looking forward to driving this

forward

thank you michelle and thomas