SREcon24 Americas - 20 Years of SRE: Highs and Lows

Summary

The speaker reflects on the evolution of Site Reliability Engineering (SRE) over 20 years, emphasizing its roots in startups and engineering solutions to operational challenges. They discuss the growth of SRE, its integration into various sectors, and the importance of its principles in preventing burnout and toil. The talk also addresses the challenges faced by SRE, including career pipeline issues, adoption failures, and the persistent perception of operations as a low-status function, advocating for a continued application of software techniques to operations.

Takeaways

• 📚 The script reflects on the evolution of the Site Reliability Engineering (SRE) role over the past 20 years, emphasizing the importance of reliability in AI and the tech industry.
• 🎨 The speaker appreciates the artistic representation of '20 years of SRE' in the style of an ancient Irish medieval illuminated manuscript, highlighting the value of creativity in technical fields.
• 🗣️ The speaker identifies themselves as a key figure in popularizing SRE, with their book on the subject being widely recognized and available in commercial spaces.
• 🌟 The talk is dedicated to the speaker's stepmother, Helen Gray, who recently passed away, adding a personal touch to the professional discussion.
• 🔍 The speaker clarifies that their perspective on SRE is personal and may not cover all aspects or align with every professional's view, urging the audience to consider this limited viewpoint.
• 🐘 The 'Elephant in the Server Room' is identified as Google, indicating the company's significant influence on the SRE field and its practices.
• 🚀 The script challenges the narrative that SRE is incompatible with the fast-paced, resource-strapped environment of startups, suggesting that SRE principles can thrive in such conditions.
• 🛠️ SRE is portrayed as an engineering-driven field, where solutions are iteratively improved upon, rather than being seen as a static, large-scale operation.
• 🌐 The speaker discusses the widespread adoption of SRE principles across various sectors and company sizes, indicating the model's versatility and relevance.
• 📈 The growth of the SRE market is noted, with the speaker observing an increase in demand for SRE knowledge and practices, despite the availability of free resources.
• 🏆 The script highlights the impact of SRE on broader societal issues, with examples of SRE professionals making significant contributions to social causes and ethical considerations in technology.

Q & A

What is the significance of the '20 years of ESS' phrase mentioned in the script?

The phrase '20 years of ESS' is used as a reflective opportunity rather than a precise timeline. It's meant to spark a discussion on the evolution and impact of Site Reliability Engineering (SRE) over the past two decades.

What is the author's connection to the popularization of SRE?

The author is responsible for popularizing SRE, largely through the publication of a book on the subject, which has been found on commercial shelves and has contributed to the widespread understanding of SRE principles.

Why is the talk dedicated to the author's stepmother, Helen Gray?

The talk is dedicated to Helen Gray as a personal tribute, acknowledging her passing and the personal significance this event holds for the author.

What does the author mean by 'the Elephant in the server room'?

The phrase 'the Elephant in the server room' is a metaphor for the obvious yet often ignored or unaddressed issue in the industry, which in this context refers to Google's influence and its relationship with SRE.

How does the author describe the evolution of Google's approach to system management?

The author describes Google's evolution as moving from a simple list of machines to more complex systems like babysitter, Borg, and others, emphasizing the importance of incremental improvements and engineering solutions to manage system reliability.

What is the author's view on the relationship between SRE and startups?

The author believes that SRE is often misunderstood as being incompatible with the fast-paced, resource-constrained environment of startups. However, he argues that SRE principles can be effectively applied in startups, where the focus is on incremental improvements and engineering solutions to problems.

What is the author's perspective on the adoption of SRE across different sectors?

The author notes that SRE has permeated various sectors, including entertainment, food delivery, education, and government, indicating a broad acceptance and application of SRE principles beyond just large multinational corporations.

Why does the author mention the book 'Site Reliability Engineering' continues to sell well?

The author points out that despite the content of the book being freely available, its continued sales indicate a demand for high-quality, curated information on SRE, suggesting that the model and practices it discusses are still relevant and sought after.

What does the author suggest about the impact of SRE on general engineering and business consciousness?

The author suggests that SRE ideas have not only permeated general engineering practices but have also influenced business consciousness, as evidenced by references to SRE in business strategy reports from prominent organizations like Gartner and Forrester.

How does the author address the issue of SRE's role in social and ethical contexts?

The author highlights instances where SRE professionals have used their skills to address broader social issues, such as the US healthcare system and the #MeToo movement, emphasizing that SRE is not just about technical solutions but also about ethical responsibility and societal impact.

What challenges does the author identify in the SRE career pipeline?

The author identifies challenges in the SRE career pipeline, particularly for junior level professionals, suggesting that the field can be intimidating and that more needs to be done to encourage and facilitate entry-level participation in SRE roles.

What is the author's view on the need for quantitative models in SRE?

The author believes there is a need for more quantitative models in SRE to provide a numeric framework for understanding the value of SRE work and to demonstrate the impact of SRE practices on organizational success.

What does the author consider the most urgent problem facing SRE today?

The author considers the persistent idea that operations, and by extension SRE, is of low status as the most urgent problem. This perception can hinder investment in reliability and user experience, despite their proven value.

How does the author summarize the impact of SRE over the past 20 years?

The author summarizes the impact of SRE as the radical idea that it is legitimate to apply software techniques and systems thinking to operations, an idea that has been revolutionary but remains surprisingly radical even in 2024.

Outlines

00:00

🎉 Reflecting on SRE's Evolution and Personal Dedication

The speaker begins by acknowledging the 20-year milestone of Error-Correcting Code (ECC), using it as a moment for reflection rather than focusing on the accuracy of the timeframe. They mention the importance of reliability in AI, as illustrated by a personal anecdote and a preference for a medieval manuscript style over a previous attempt. The talk is dedicated to the speaker's stepmother, Helen Gray, who passed away recently. The speaker clarifies that their perspective is personal and may not cover everyone's favorite aspects of SRE due to limited view. They introduce the concept of the 'Elephant in the Server Room,' a metaphor for an obvious issue that is being ignored, which in this case is Google's influence on SRE. The speaker also discusses the evolution of the SRE role and the perception changes, challenging the narrative that Google's size and influence make SRE inflexible and machine-like. Instead, they propose an alternative view, considering Google's startup origins and the SRE's adaptability and engineering mindset.

05:01

🔧 The Engineering Mindset Behind SRE's Growth

The speaker delves into the engineering mindset that has driven SRE's growth, emphasizing the iterative process of replacing 'slightly less terrible' solutions with better ones. They recount the early days at Google, where the need for managing clusters led to the creation of 'babysitter,' a system that, despite its flaws, was an improvement over the previous ad-hoc methods. The speaker argues that SRE's success is rooted in this engineering approach, which focuses on incremental improvements and adaptability, rather than being a static, large-scale operation. They also touch on the importance of storytelling in shaping the understanding of SRE's role and history, highlighting the need for accurate narratives that reflect the reality of SRE's dynamic and problem-solving nature.

10:01

🌐 SRE's Broad Impact and Integration into Various Sectors

The speaker highlights the widespread adoption of SRE principles across various sectors, from entertainment to government, indicating the model's versatility and relevance. They note the continued interest in SRE material, as evidenced by the ongoing sales of related books, despite the content being freely available. This suggests a demand for high-quality information on the subject. The speaker also observes that SRE concepts are not only permeating general engineering practices but also influencing business consciousness, as seen in the attention from prominent IT strategy organizations like Gartner and Forrester. They emphasize the importance of SRE's contribution to engineering and societal contexts, citing examples of SRE professionals making significant impacts beyond their roles, such as influencing healthcare systems and initiating social movements.

15:03

🛠️ SRE's Role in Engineering and Social Change

The speaker discusses the significant engineering contributions of SRE teams over the past two decades, noting their vital involvement in the development of successful Google systems that have had a broad impact, both internally and externally through citations in academic papers. They also reflect on SRE's social influence, with professionals from the field appearing on Time magazine covers for their roles in fixing broken systems and initiating change. The speaker ponders the reasons behind SRE's inclination towards positive social impact, suggesting it may be due to the profession's focus on user experience, a strong mission orientation, or the historical influence of early SRE professionals. They conclude that SRE's ability to prevent burnout and toil through its practices is a significant achievement in itself.

20:04

🚧 Challenges and Failures in SRE's Adoption and Identity

The speaker addresses the challenges faced by SRE, particularly concerning the career pipeline for junior level professionals and the intimidation factor for outsiders. They express concern over the failures in SRE program adoption within companies that should be prime candidates for such practices. The speaker regrets the lack of organizational post-mortems to understand these failures better. They also touch on the dilution of the SRE term and the potential overemphasis on identity within the profession. The speaker advocates for a focus on generating more models, especially quantitative ones, to understand and communicate the value of SRE work better. They highlight the need to counteract the idea that reliability no longer matters due to economic shifts and to continue investing in user experience.

25:06

🌟 The Radical Notion of Applying Software Techniques to Operations

In the concluding remarks, the speaker reflects on the past 20 years of SRE and the industry's current state. They reiterate the radical idea that it is entirely legitimate to apply software techniques and systems thinking to the operations domain. Despite the progress made, the speaker asserts that this concept remains radical in 2024. They encourage challenging the notion that operations are of low status and emphasize the societal bias against maintaining systems and executing patterns. The speaker ends on a reflective note, acknowledging the significant journey SRE has undertaken and the need to continue pushing for its recognition and integration in various domains.

Keywords

💡SRE (Site Reliability Engineering)

SRE, or Site Reliability Engineering, is a discipline that incorporates aspects of software engineering to improve the reliability, scalability, and performance of production systems. It is central to the video's theme as it discusses the evolution and impact of SRE over the past 20 years. The script mentions how SRE has grown from a concept to a widely adopted practice across various industries, with the speaker reflecting on its beginnings and its role in handling system management and automation.

💡Reliability

Reliability in the context of the video refers to the dependability of a system or service to perform its intended function without failure. It is a core aspect of SRE, where the goal is to ensure that systems remain available and consistent. The script uses the phrase '20 years of ESS' to reflect on the history of SRE's focus on reliability, and how it has been a driving factor in the development of systems and practices within Google and beyond.

💡Startups

Startups are new businesses that are typically characterized by limited resources, rapid growth, and a high level of uncertainty. The script discusses the relationship between startups and SRE, highlighting that Google, like many startups, had to navigate through relentless pressure and change. The speaker argues against the perception that SRE is incompatible with the fast-paced, chaotic environment of startups, suggesting that SRE principles can be effectively applied in such settings.

💡Engineering

Engineering in this video is presented as the process of creating solutions to problems, often through the application of scientific, economic, social, and practical knowledge. The script emphasizes the engineering mindset of SREs, who are not just maintaining systems but actively working to improve them. The speaker shares anecdotes about how SREs at Google engineered their way out of problems, leading to the development of systems like babysitter, Borg, and others.

💡Toil

Toil in the video refers to the manual, repetitive, and often tedious work that can lead to burnout and inefficiency. The speaker discusses the cursed knowledge that SREs have gained from understanding that much of the toil associated with system management is preventable. The script suggests that one of SRE's achievements is helping the industry recognize and address the issue of toil.

💡Burnout

Burnout is a state of chronic stress that can result from overwork and lack of balance between work and personal life. The video addresses the issue of burnout in the context of SRE work, suggesting that the profession has made strides in understanding and preventing it. The script implies that by applying SRE principles, the industry can mitigate the risk of burnout for those involved in system management.

💡Evolution

Evolution in the video refers to the development and changes that have occurred in the SRE field over the past two decades. The speaker reflects on how SRE has evolved from its early days at Google to become a recognized discipline within the tech industry. The script discusses the evolution of SRE roles, practices, and the perception of the field.

💡Postmortem

A postmortem in the context of the video is an analysis conducted after a significant event, such as a system failure, to understand what happened and how to prevent similar occurrences in the future. The script mentions the importance of postmortems in SRE for learning and improvement, and the speaker expresses a desire for more organizational post-mortems related to SRE adoption failures.

💡Career Pipeline

The career pipeline refers to the progression of career stages and opportunities within a profession. The video discusses the challenges of the SRE career pipeline, particularly for junior level individuals. The script suggests that the profession needs to do more to attract and develop talent, ensuring that those entering the field can grow and advance.

💡Quantitative Models

Quantitative models in the video are numerical frameworks used to measure and understand complex phenomena. The speaker calls for more quantitative models in SRE to better understand and articulate the value of the work done by SREs. The script implies that such models could help in demonstrating the impact of SRE practices on system reliability and user experience.

💡Operations

Operations in the video refers to the ongoing processes and systems that support the functioning of a business or service. The speaker discusses the persistent undervaluation of operations within the industry, suggesting that there is a societal bias against investing in the maintenance and reliability of systems. The script argues for a change in perception to recognize the importance of operations and the role of SRE within it.

Highlights

20 years of ESS reflection without fixating on the exact phrase, using it as an opportunity to look back on the past.

The importance of reliability in AI illustrated by comparing two different renderings of '20 years of ESS'.

The speaker's role in popularizing SRE through his book, visible on commercial shelves.

Personal dedication of the talk to the speaker's stepmother Helen Gray, who passed away last month.

Clarification that the talk represents a personal view, not influenced by any multinational or department.

The 'Elephant in the server room' analogy for the often overlooked entity of Google in SRE discussions.

Contrasting the perception of Google as a relentless, infallible giant with the reality of its startup origins.

The misconception that SRE is incompatible with the fast-paced, chaotic nature of startups.

The narrative of SRE evolving from the need to engineer solutions to immediate problems in startups.

The story of 'babysitter', an early system at Google that, despite its flaws, was a step up from previous methods.

The role of SRE in the development of Google's production systems like Borg and work q.

The idea that SRE principles have permeated general engineering and business consciousness.

The observation that SRE practices are being adopted across various sectors and company sizes.

The ongoing success and demand for SRE material, despite it being available for free.

The contribution of SREs to significant Google systems and their wider industry impact through publications.

SRE's role in fixing broken systems and its connection to ethical considerations and social impact.

Concerns about the SRE career pipeline, particularly for junior level professionals.

The issue of SRE program adoption failures and the lack of industry post-mortems to learn from them.

The need for more quantitative models to understand the value and impact of SRE work.

The challenge of maintaining the relevance of SRE in a changing economic climate and the importance of user experience.

The persistent societal bias against operations work and the low status perception of operational roles.

The radical idea that software techniques and systems thinking can be applied to the operations domain, still considered radical after 20 years.

Transcripts

[Music]

so first things first starting with the

title 20 years of ESS long time let's

not fixate on the accuracy of that

particular phrase uh I think we should

use it as an opportunity to reflect on

the past and in the spirit of this and

also in the spirit of the fact that s

Recon Americas makes me miss my national

holiday St Patrick's Day March the 17th

every single year here is mid Journey's

rendering of 20 years of ESS in the

style of a ancient Irish medieval

illuminated manuscript which I greatly

prefer to the uh rendering of the first

attempt uh which I think nicely

illustrates the importance of

reliability in uh a I okay so uh we've

got a lot to cover 20 years in 20

minutes plus questions so by linear

extrapolation that's one year per minute

uh so I'll talk very quickly obviously

uh who am I and why should you care uh I

suppose I'm probably more responsible

than most for popularizing s in terms of

this uh this book which you can see in

the picture on the left I am very happy

to find on the commercial shelves of a

Bookshop somewhere down in the value

actually uh so seen a few things uh

there's a picture of me ripping it up

which is part of a previous s Recon

opening plary if I recall correctly

here's a picture of me writing another

one the other photo on a personal note

is my stepmother Helen Gray who passed

away last month and to whom this talk is

dedicated and on that personal note I

want to emphasize that this is a

personal view of a talk right so there's

no giant multinational there's no PE

Department P department is me so in

other words if I'm missing your favorite

thing it's because of my limited view

and limited perspective so just want to

make that clear but with all of those

preliminaries out of the way anyone take

a guess what this

is elephant in the server room it's the

elephant in the server room accid

waiting to

happen so popular English phrase meaning

the obvious thing known as talking B

and the Elephant in the server room is

of course

Google the uh image here is a internal

meme that got leaked to BuzzFeed if I

recall correctly so there's going to be

a certain amount to Google in this

presentation you are unfortunately going

to have to be okay with that but I began

my Reflections on 20 years by thinking

about the evolution of the role and the

change in perceptions of this role and I

realized that I kind of have an internal

story about what s is and how it's

changed which is different to many other

people and in fact I think there's one

key story which has grown up around

Google and relationship with s that I

think it's necessary to attack actually

um so I appreciate this may be shocking

to you but sometimes stories aren't

true or different sets of stories are

available depending on the evidence you

select from so I'll take one story to

start with which is that is Google is

smooth and Relentless and infallible and

gigantic and so on and so forth and S as

a result inherits those properties and

so we'd think of s as kind of having a

machine like quality right so giant

resources giant machine giant problems

so

on but other stories are

available to uh do so I'll have to talk

a little bit about startups so startups

what are startups like startups are like

eating glass as Shawn Parker and a

number of other people said various

other entrepreneurs have said uh

startups have a lot of characteristics

that make them kind of not really like

ordinary life so there isn't really a

steady state things are changing all the

time and if they aren't that's bad

Relentless pressure and so on the

shocking news or the different story is

that Google was a startup once as well

and so one thing that's kind of

developed over time is that people seem

to think that s is somehow in opposition

to or incompatible with constant change

fast moving startups in general and so

on I think there's a bunch of reasons

that people believe this but I want to

hone in on one for brevity's sake might

not be your favorite reason but it is

this that people seem to think that

startup chaos Relentless pressure and

not enough resources being

incompatible is function of I I

suppose Clarity or the the the the

pressure to produce means that you

output bad Solutions because you haven't

got the time to do better Solutions and

so now that I'm saying that finishing

your slides five minutes before you get

on stage produces your best work all of

the

time uh necessary short-term Clarity

around choices has produced some of my

best work although for the sake of

engineering uh coherence I feel I should

tell you it has also produced some of my

worst

work but there is this conflict right

and uh I

think it's curious to me to hear how the

story of ere has become for want of a

better term very big company because in

my experience one very real Loop that

drives work in startups is the

following which is putting the slightly

less terrible thing in place before

tomorrow's terrible thing

happens and so the way you get from one

of these things to another is by

engineering and in my opinion in my

story SRE grows from this not as a break

on progress not as a necessary component

of a planet spanning multinational not

as an infinitely large smoothly working

machine but you're doing things while

they're useful then when they're not you

stop doing them you start doing

different things

instead the story is really driven by

engineering and so let me tell you a

story about the early early days which I

was not present for all of uh to be

clear so I am relying on on research and

stories but there was a file at one

stage called cluster.

py Google production starts off very

very simply there's essentially a list

of machines and specific software runs

on specific machines so there's AR raay

defined in some python somewhere and

it's like okay the indexing is on

machine one comma machine 3 comma

whatever okay fine so it turns out

machines go down change rol so on and so

forth and how do people handle this well

they Fork the file and add their own

stuff until eventually there's one fork

for every

FTE or well not quite that bad but you

know close to that So eventually

something better is needed and can't buy

anything to do this right it's all

internal stuff so there's a system built

called I think babysitter that has been

talked about publicly uh and it's really

based around the astonishing realization

that if you can provide some mechanism

to change the arrays at runtime you

don't actually need to Fork the script

anyway very successful so this is the

key Point babysitter is terrible but

it's incrementally less terrible than

the thing that was before it and so in

due course it it buys us time to put in

other things work q and Borg and a bunch

of other kind of systems come along and

Sol

various uh components of the cluster

scheduling

workload and so I would say that

S as an organization would be centrally

involved in this but actually this is

too early like s doesn't even exist at

this point but s thinking

does there's a set of people who care

about production no matter what they're

called and their inclination is to

engineer their way out of problems often

engineering directly into more problems

but definitely engineering their way out

of

problems

okay so I've been talking a lot about

stories playing a role in how you

understand yourself obviously every good

superhero needs an origin story alas so

do the evil ones one uh is the one of a

better term official story so Ben

trainer sloth does a great job of

organizational alignment and

storytelling and so on uh following on

from the work uh I discovered my

research of ma Bloomfield who's manager

of the very earliest production teams in

Google wanted to mention her name

publicly but other stories are available

other selections from the evidence are

available item two is another story h i

it's not as well sourced right so it

basically says is that from very early

on there is a sense that Google doesn't

want to hire folks who are just going to

follow playbooks for the act of system

management want people who could

actually write software automate the

playbooks etc etc I can't verify this I

have been told it I think it does play

into the situational awareness for toil

that Google SRE in particular has uh and

that maybe not a lot of other places

have actually uh but three is kind of

what I've been talking about what I want

to surface which is I think Beyond any

confusion around naming or title or role

or any of that stuff is the core facts

that a when you get people who care

about production and B when they're

allowed to when they can engineer stuff

with to in production you get magic

happening and so when I reflect back on

the gigantic progress of SRE over the 20

years which is actually like it is

really something this thing we do from

the before the earliest days that it had

this name to this gigantic thing it's

become and how utterly bizarre that

seems in retrospect I try to hang on to

those two facts A and B to keep me

grounded okay uh prizing over let's move

on from history somewhat uh and earliest

days to themes and ideas highlights and

low lights we can pick out of the course

of

history uh so let's start out with the

obvious One S teams are everywhere this

set of logos that I stuck up there is

absolutely not a definitive list it is

the list of places that had jobs open

when on Saturday I opened the tab and

did the

search but it gives you a flavor of the

places that s has made its way into and

so from a sectoral point of view it's

like entertainment food delivery

education retail government trading

driverless

cards I also have reason to believe that

according to conference data at least s

is not the sole province of Planet

spanning multinationals but has made its

way into smmes mid-range companies and

so on the middle range having ticken Deb

sufficient uh significantly overtime

which I think is a good sign for the

profession so

s material continues to sell well I am

not looking happy because of this uh I

make no money from this book uh why is

S3 material continuing to sell well

actually the model is new and I know you

are sitting in your chair going well

actually it's 20 years old you just told

us of course the industry is large and

it takes a lot of time for ideas to

propagate and models in particular even

if it's only the top tier you care about

but I would say that if your world is

just devops or iil or platform or

whatever there is a genuine Gap into

which SRE can fit there is a legitimate

oh excellent will this collapse

underneath me no it won't I'll move over

here that'll make it better uh so if

your world is devops iil platform enge

Etc the SR model does have something to

say to you about how systems should be

managed how organizations should be run

or structured so on I think net net the

market continues to grow and people want

to continue to learn about it I mean

that's why you're here that's why

there's a huge proportion of new folks

which I was very interested to to

see uh but also and this is a bit weird

but it is true the content of these

books is available for free and yet yet

I know they still get bought they still

get read online in various ways that

actually cost money as opposed to not

costing money so that suggests to me

that there's still significant demand

for what's regarded as high quality

information so uh this book's title for

those who can't read it is okay let's do

your stupid idea which is the implicit

title of a number of engineering

conversations I have been in

the general point I am trying to make is

that there's a ton of ideas about how to

understand the problem of software and

systems management creation design and

so on outlined in our body of

knowledge and it would be one set of

consequences for the universe if that

knowledge was kind of scoped to us and

jealously guarded and so on so forth but

actually I don't think it's that way I

think it has

escaped and it's obviously hard to come

up with objective measures for this in

some sense but hey I've got 20 minutes

so we're on to anecdotes at the

frequency I have noticed of explaining

certain ESS ideas to kind of shall we

say generic software Engineers or

product Engineers uh has dropped so I

find that I don't have to Define for

example slos as often as I did several

years ago Etc the knowledge is permeated

getting out there another thing I see is

that cross referencing to ESS ideas

whether in the form of the book or

papers or talks or whatever is also

increasing over time I do think there's

room for science in uh in looking at

this but certainly my experience is that

these two effects are are definitely

ongoing but as well as s

ideas permeating General

engineering I think also some of them

have made their way into General

business Consciousness which is also

pretty weird so uh Gartner and farer for

those who don't know are kind of very

large and important uh it organizations

it strategy organizations kind of might

have heard of the Gartner magic qu

quadrant and a bunch of other things

anyway they tell other organizations

what to think about

it and one of the consequences of the

massive popularity the thing that we're

doing is that the Machinery of business

sat up and took notice and uh I won't

use the phrase churns out but writes a

lot of stuff about SRE and hence you see

articles like this from Forester and

claims like this from

Gartner uh as I say both well-known and

extremely Salient

organizations I think in particular I

suppose you you could query the number

numbers in the Gartner statement that by

2027 75% of enter Enterprises will use

ESS practices up from 10% in 22 but I

think whatever about the the state of

those numbers s has created that

conversation has revolutionized that

conversation like we're not having that

that sentence isn't written in a world

where s doesn't

happen and I think it's also interesting

to note that uh this Forest article from

2018 doesn't conclude 46% of them apply

of Google's s principles apply directly

to your Enterprise what about the rest

no they're terrible don't use them it

concludes that a bunch of the rest are

also applicable under different kind of

circumstances so in some sense the IT

industry has looked at this and has

picked the things that apply and has has

found that it it actually works for them

uh yeah I think another point I would

say just hammering on the engineering

piece again

is uh s teams over the past 20 years

have got a lot of engineering

done again this is my personal

experience my very limited view my

myopia but as an example here are two

papers describing Google systems that I

had sight of what the SRE contribution

was and it was vital to both of them and

both of those systems were very

successful internally I think might even

I'm I'm pretty sure the one on the right

is still working and if the one on the

left isn't working bad things are

happening right now um externally like

these papers have gone on to be cited

hundreds of times and like there's

there's huge impact from these things um

I highlight the SRE contributor names as

I remember them but of course I might

forget them etc etc

so again from my personal point of

view it's also been great to see so many

sres doing work that benefits others

aware of a wider kind of social context

um or again for want of a better term

ethical I'm actually fascinated by this

and I've thought about it a lot over the

years like why why is this is it because

that S as a

profession kind of thinks about the user

or is geared around the positive

experience of somebody else on a

day-to-day basis or is it because of the

strong sense of above and beyond or

Mission

orientation that Google s certainly had

early on or maybe the historical

accident of a bunch of people who joined

early on either way it is notable to me

at least that for such a young

profession we've already had two Time

magazine covers with s people on them on

the left the us Healthcare System such

as it was rescued by SES I think Mikey

Dickerson who you may hear from later on

uh goes on to lead a substantial

government Department to try and do more

of the same and on the right it's Susan

Fowler ESS at Uber whose memo helps to

kick off the me2 movement uh and I think

again for me for people who feel s has

nothing to do with this uh that actually

fixing broken systems and fixing broken

organiz ational systems and

organizational postmortem are very

definitely part of what s does I

surfaced these two examples because

they're convenient graphically I could

mention others uh Liz Fong Jones and

kiwi farms and a bunch of other stuff

prominently again for my limited

perspective uh I would also note that

when um generic software Engineers have

made it to the cover of time it's not

necessarily because of the benefit to

other

people

so continuing on that topic

uh I just want to explore a little bit

more in the context of this quote of a

review of the ESS book that Chris Jones

pointed me out

recently which is the ESS book has

granted me the cursed knowledge cursed

knowledge to see that the misery of toil

and burnout misery sorry is entirely

preventable and I think s at its best

has a long history of doing things

different L because it is the right

thing to do not because we did it that

way previously or we're always going to

do it this way or

whatever but here if the only thing we

did as a profession if literally the

only thing we accomplished was to help

the industry understand that

actually it might be cursed knowledge

but actually burn out and toil and so on

are preventable it would be worth it on

its

own now

okay low lights so uh in true Centrist

dad fashion time for the low lights uh

first thing I will start with is the

career

pipeline the career pipeline I think is

particularly problematic in SRE for

junior level folks uh in fact I'm aware

of one org where s it was suggested s

should only be defined for senior level

and up H and I've said elsewhere that

one close analog for S is Staff engineer

like those kind of crosscutting concerns

that you often see staff Engineers

blessed with uh are can resonate in the

S world as well but if we can't ingest

Folks at the bottom of the profession

and get them to the top and if we can't

take all of the folks who want to do

that then we have some trouble and S is

intimidating from the outside I mean

despite the wealth of educational

material available we should do more

about

that uh this was a ATM in New Orleans

where I refrained from pushing either of

those buttons though I was

tempted device initialize in particular

uh it is useful it's useful balance I

think to say that as well as the success

in adoption that we've outlined earlier

we've also seen plenty of failures in s

program adoption and in companies that

would normally be considered kind of

fertile ground for this kind kind of

thing so large well resourced

technically capable a lot at stake and

so on my personal regret from these is

not that the adoption failed but that

it's not in the nature of the industry

to write organizational post-mortems and

I would love to see uh love to see those

for the adoption

failures it is also kind of unpopular

puffing opinion of me uh but as well as

the genuine problems that are caused by

the mishandling of the term s dilution

of what it means problems with the

implementation and so on we worry about

that a lot in my opinion possibly worry

about it identity too much maybe we

should do less of

that in

particular I've I've spoken about this

in detail in other kind of talks uh

including other s Keynotes so I won't

labor this point um but I will say that

I I don't feel we're really generating

enough models about what it is that we

are doing or trying to do um I

specifically highlight mathematical

there uh because I do think we have a

lot of qualitative models we could

always use more and that's totally fine

um but I I think we need some more

quantitative models about what it is

we're doing and in particular

quantitative models about being able to

put generic numeric Frameworks on the

value of what it is we do on how much

our work matters and I think my personal

intuition is there's a lot of Runway for

this and we just aren't really getting

started uh it's our most urgent problem

in my

opinion uh you will be familiar with the

fact that zero interest rates have gone

away my mortgage certainly is very

familiar with that fact thank you uh it

has changed a bunch of behaviors in

companies aligning with layoffs

attitudes to expenditure for operations

keeping things going and so on again

I've spoken about this before won't

embellish it but I will say that that

attacking the ideas that reliability no

longer matters because growth is

unavailable which is an argument I've

heard and because we can't put a good

number on the worth of the user

experience we won't invest in it those

are kind of key ideas to attack and for

what it's worth I say abstract economic

bases there because I'm pretty sure the

customers still want their stuff to work

whatever the abstract economic rationale

is

is and I think also there are several

patterns which are pretty prevalent in

human behavior and one of those is the

tendency to confuse Doctrine with

identity and so it's particularly

natural if your world has been

significantly

disrupted uh you may end up clinging to

to Doctrine in some sense if you don't

have an easy replacement to

hand but my final low light and I do

think it's an important one is

underpinning some of the earlier low

lights at least at decision maker level

is the persistent idea that operations

is low

status so of course worth is socially

constructed to a large

extent and it's also true that we have a

societal bias against keeping things

going or kind of paying for keeping

things going a bias a kind of

intellectual viice in some sense against

good pattern execution rather than

creation of new patterns and so on and I

can stand here and say that I think we

need to fix this which is

true but in a talk which is mostly about

reflection I'm thinking about the

relationship with s and the past 20

years and all of that does Put Me In

Mind of one potential summary of the

past 20

years which is towi the radical idea

that it is legitimate to apply software

techniques and systems thinking to the

operations

domain and in 2024 I stand here and I

look at the past 20 years and I look at

the state of the industry in the state

of the world and I say that that idea is

still

radical thanks for your time come and

say up

boo thank you very much