History's Worst Software Error

Summary

The script recounts the tragic history of the Therac-25, a radiation therapy machine that caused severe injuries and deaths due to software errors. It highlights the importance of rigorous software testing and ethical decision-making in medical technology. The Therac-25's malfunctions, including a deadly bug known as 'Malfunction 54', led to the machine's withdrawal and the dissolution of the manufacturer's medical division.

Takeaways

• 😔 Katie Yarborough was the first victim of the Therac-25, a radiation therapy machine with fatal software flaws.
• 🔬 Therac-25 was a state-of-the-art linear accelerator designed for cancer treatment, but it was primarily controlled by software.
• 💡 The concept of radiotherapy involves using high-energy radiation to target and destroy cancer cells.
• 🛠️ Therac-25 was developed by AECL Medical and was smaller and more software-dependent than its predecessors.
• 🚫 A critical oversight in the 1983 safety analysis excluded any evaluation of the Therac-25's software.
• 🔍 The software, based on the older Therac-20 model and written by a single individual, had significant errors that were not addressed.
• 🚨 Multiple patients were severely injured or killed due to software malfunctions that caused overdoses of radiation.
• 🤖 Therac-25's software had a critical flaw known as 'Malfunction 54', which allowed unfiltered radiation beams to hit patients.
• 🛑 AECL initially denied the possibility of overdoses and did not acknowledge the software issues until forced by evidence and lawsuits.
• 🔄 Therac-25's design relied on software without hardware fail-safes, leading to a series of tragic accidents.
• 📚 The Therac-25 case is now a cautionary tale in ethics and computer science, highlighting the importance of rigorous software testing and safety measures.

Q & A

What was the name of the woman who experienced the first known software-related accident in the medical field?

Katie Yarborough.

What was the medical device involved in the accidents described in the script?

The device involved was the Therac-25, a linear accelerator used for cancer treatment.

What was the year when Katie Yarborough's accident with the Therac-25 occurred?

Katie Yarborough's accident occurred in 1985.

How did the Therac-25 work in terms of delivering radiation to patients?

The Therac-25 worked by using a double pass accelerator to direct high-energy electrons and/or X-rays into patients' lymph nodes.

What was the estimated radiation dose Katie Yarborough received instead of the prescribed 200 rads?

Katie Yarborough received an estimated dose of 20,000 rads, which was hundreds of times more than the prescribed amount.

What was the name of the company that developed the Therac-25?

The Therac-25 was developed by AECL Medical, a division of Atomic Energy of Canada Limited.

What was the main issue with the Therac-25's software that led to the accidents?

The main issue with the Therac-25's software was the lack of proper safety checks and the potential for errors like arithmetic overflow, which allowed unfiltered beams of radiation to strike patients.

What was the 'Malfunction 54' error that was repeatedly mentioned in the script?

Malfunction 54 was an undefined error in the Therac-25's software that, when encountered, allowed the machine to deliver a powerful, unfiltered beam of radiation to patients, causing severe injuries or death.

What was the corrective action plan (CAP) that AECL eventually submitted to the FDA after the accidents?

The corrective action plan included 23 software changes and six hardware safety features, including a dose per pulse monitor to shut down dangerous doses even if all software safety checks failed.

What was the final outcome for AECL Medical after the Therac-25 accidents?

AECL Medical dissolved their medical division in 1988, and lawsuits from the families of the victims were settled out of court.

How is the Therac-25 incident viewed today in the context of medical technology and ethics?

Today, the Therac-25 incident is considered a staple of ethics and computer science classes as a case study of what can go wrong when new technology is trusted implicitly and ethical decision-making fails.

Outlines

00:00

😨 The Tragic Beginning of the Therac-25 Story

The script begins with the tragic story of Katie Yarborough, a 61-year-old manicurist who underwent her 12th cancer treatment using the Therac-25 at Kennestone Regional Oncology Center. Instead of painlessly receiving 200 rads of radiation, she experienced a severe burn. Despite the technician's reassurances, Yarborough suffered extensive damage, including the need for a mastectomy and paralysis of her left arm. The incident marked the beginning of what would be recognized as some of the worst software-related accidents in history. The script then explains the concept of radiotherapy and its evolution into a sophisticated field using advanced imaging and targeted radiation beams. The Therac-25, developed by AECL Medical, was a revolutionary machine that used software to streamline radiation delivery. However, the safety analysis conducted in 1983 overlooked the software's potential flaws, which were inherited from the older Therac-20 model and written by an unidentified coding hobbyist.

05:02

🤕 The Escalation of Therac-25 Malfunctions and Their Consequences

The script continues to describe the escalating series of malfunctions with the Therac-25, which resulted in severe radiation overdoses for patients. It recounts the experiences of a 40-year-old woman with cervical cancer who suffered a similar fate to Yarborough, and the subsequent investigation by AECL, which initially dismissed the possibility of an overdose. The narrative details how the Therac-25's software was error-prone, with operators encountering multiple error messages daily and pressing 'proceed' without fully understanding the implications. The script also describes the tragic cases of other patients, including Voyn Ray Cox and Vernon Kidd, who died as a result of the machine's malfunctions. It highlights the challenges in identifying the root cause of the malfunctions, particularly 'Malfunction 54,' and the slow response from AECL to address the issues.

10:05

🔍 Unraveling the Mystery of Malfunction 54

This section delves into the investigation of Malfunction 54, which was responsible for delivering lethal doses of radiation to patients. The script describes how the Therac-25's software and hardware were designed without adequate safety checks, allowing for the possibility of unfiltered radiation beams to strike patients. It explains the eight-second delay in the machine's magnets moving into position and how operator actions within this timeframe could lead to critical errors. The script recounts the efforts of Dr. Fritz Hager and a technician to recreate Malfunction 54, ultimately discovering the issue and prompting the FDA to declare the Therac-25 defective and demand corrective action from AECL.

15:07

🚫 The Continuing Tragedy and the Final Resolution

Despite identifying and attempting to correct Malfunction 54, the script reveals that accidents with the Therac-25 continued, resulting in further fatalities. It discusses the discovery of another software error related to an 'overflow' condition that led to the death of Glenn Dodd. The FDA once again declared the Therac-25 defective and demanded all units be taken out of service. The script outlines the final corrective action plan submitted by AECL, which included significant software and hardware changes, and the eventual dissolution of AECL's medical division in 1988. The Therac-25 is now remembered as a cautionary tale in ethics and computer science classes, highlighting the dangers of trusting new technology without proper oversight and testing.

20:09

🏥 The Legacy of the Therac-25 and Lessons Learned

The final paragraph reflects on the legacy of the Therac-25 and the lessons learned from its tragic history. The script emphasizes the importance of ethical decision-making in technology development and the need for rigorous testing and documentation of medical software. It also notes the changes in regulatory requirements by the FDA for new medical products, which now include independent investigation of software documentation. The script concludes with a quote from the lawyer for the first Therac victim, highlighting the irony of such sophisticated machines causing harm due to a lack of common sense in their operation and safety measures.

Keywords

💡Therac-25

The Therac-25 was a radiation therapy machine developed by AECL Medical, a division of Atomic Energy of Canada Limited. It is central to the video's theme as it represents the tragic consequences of software errors in medical technology. The Therac-25 is infamous for causing severe injuries and deaths due to malfunctions, which were later attributed to software bugs, as depicted in the video through various case studies.

💡Radiation Therapy

Radiation therapy, also known as radiotherapy, is a medical treatment that uses high-energy particles or photons to destroy cancer cells. It is a key concept in the video as it explains the intended use of the Therac-25. The script describes how radiation can ionize atoms in DNA, leading to cell death, which is a fundamental principle in fighting cancer without invasive surgeries.

💡Software Malfunction

A software malfunction refers to an unintended behavior or error in the operation of a software program. In the context of the video, the term is pivotal as it discusses how software malfunctions in the Therac-25 led to overexposure of patients to radiation, resulting in severe injuries and fatalities. The video details specific malfunctions, such as 'Malfunction 54,' which caused the machine to deliver lethal doses of radiation.

💡Linear Accelerator

A linear accelerator is a device that produces high-energy beams of particles, commonly used in radiation therapy. The Therac-25 was a type of linear accelerator that used a double pass accelerator design. The video explains how this technology was intended to streamline the delivery of radiation but was ultimately compromised by software issues.

💡High-Energy Radiation

High-energy radiation refers to electromagnetic waves or particles that carry significant energy and can penetrate materials, including human tissue. In the video, high-energy radiation is the therapeutic agent used by the Therac-25 to target cancer cells. However, the video also highlights the dangers of this technology when software errors lead to miscalculations in dosage and targeting.

💡Cancer Treatment

Cancer treatment encompasses the various medical procedures aimed at eliminating cancer cells or controlling their growth. The video discusses the use of the Therac-25 in cancer treatment, specifically radiation therapy. It also touches on the devastating impact when the treatment itself becomes harmful due to software errors, as seen in the victims' stories.

💡Safety Analysis

Safety analysis is the process of evaluating a system to identify potential hazards and ensure it operates within safe parameters. The video criticizes the 1983 safety analysis of the Therac-25 for failing to include a thorough examination of the software, which later led to catastrophic accidents.

💡Arithmetic Overflow

Arithmetic overflow occurs when an arithmetic operation exceeds the storage capacity of the system's memory, causing it to wrap around to zero. In the video, this concept is used to explain one of the software errors in the Therac-25 that allowed for an underestimation of the radiation dose, leading to a fatal overdose for a patient.

💡Ethical Decision Making

Ethical decision making involves choosing a course of action that is morally and professionally appropriate. The video implies that ethical decision making was lacking in the development and oversight of the Therac-25, with the company failing to adequately address known issues and prioritize patient safety.

💡Medical Malpractice

Medical malpractice refers to professional negligence by healthcare providers that results in harm to a patient. While the video does not explicitly use this term, the concept is implied through the discussion of the Therac-25 accidents, suggesting that the combination of software errors and the company's response constituted malpractice.

💡Corrective Action Plan (CAP)

A Corrective Action Plan (CAP) is a strategy to identify, document, and eliminate the root causes of a problem. The video describes how AECL developed a CAP in response to the FDA's declaration of the Therac-25 as defective, which included both software and hardware changes to address the identified malfunctions.

Highlights

Katie Yarborough's tragic experience with the Therac-25, marking one of the worst software-caused accidents in history.

Therac-25, a state-of-the-art linear accelerator, was used for cancer treatment by directing high-energy electrons or X-rays into patients' lymph nodes.

Katie Yarborough's severe injury from an overdose of radiation, leading to the removal of her breast and paralysis of her arm.

Therac-25's innovative double pass accelerator design by AECL Medical, which streamlined the process by sending beams through a target twice.

Therac-25 was primarily run by software, which was a significant shift from previous hardware-dependent accelerators.

Safety analysis of Therac-25 omitted software interrogation, which was based on the older Therac-20 model and written by a single unidentified coder.

The discovery of an exit dose on Yarborough, indicating an electron beam had been used, which was inconsistent with her prescribed dose.

AECL's initial denial of any overdose possibility and their request for the medical physicist to stop making unproven claims.

Subsequent similar accidents with Therac-25, indicating a systemic issue with the machine's software or operation.

The identification of 'Malfunction 54' as a critical software error that allowed unfiltered radiation beams to strike patients.

Operators' testimony of frequent error messages and their practice of proceeding without fully understanding the implications.

Therac-25's reliance on software without hardware interlocks, which removed physical safety mechanisms.

The FDA's declaration of Therac-25 as defective and the demand for a corrective action plan.

AECL's eventual acknowledgment of the software issues and the implementation of 23 software changes and six hardware safety features.

The end of Therac-25's use in hospitals and the dissolution of AECL's medical division due to the high-profile accidents.

Therac-25 now serves as a case study in ethics and computer science classes, emphasizing the importance of responsible technology use and ethical decision-making.

The FDA's current requirement for documentation on all software for new medical products, highlighting the lessons learned from Therac-25.

Transcripts

katie yarborough woke up on a warm clear

june day in 1985 and prepared for her

12th cancer treatment

the 61 year old manicurist got dressed

and drove herself to the kennestone

regional oncology center in marietta

georgia where a state-of-the-art linear

accelerator called the therac-25

would direct high-energy electrons and

or x-rays into her lymph nodes as it had

done for patients in the area thousands

of times before

the therac would need only a few seconds

to painlessly deliver around 200 rads to

her upper left chest

but that day

something went wrong

yarborough felt a red-hot sensation

instead of nothing

you burned me

she told the technician who quickly

assured her that this wasn't possible

over the next few weeks she would need

one breast fully removed and her left

arm would become completely paralyzed

but her useless arm didn't stop her from

living her life or from driving

she died five years later when her car

was struck by a truck

on a georgia highway

katie yarborough was the first victim of

what would be later called some of the

worst software caused accidents in

history

this

is the true story

of the therak-25

how do you treat an insidious and

deep-seated disease like cancer and its

many forms without invasive and

dangerous surgeries

one answer in use for over a hundred

years now is radiation or radiotherapy

the concept is simpler than its name

suggests

radiation in the form of high energy

particles and photons can ionize or

otherwise change atoms and molecules

in a sensitive structure like dna enough

of this damage can lead to the death of

a cell

a disease like cancer progresses through

the unchecked division of cells so why

not try blasting these mutants with

radiation that can by its very nature

pass invisibly through body tissue

what began in earnest only after world

war ii and the first nuclear reactors is

now a highly sophisticated field that

uses three-dimensional body imaging and

targeted beams of radiation from linear

particle accelerators to prevent

halt and otherwise destroy cancer cells

in the best cases radiotherapy is

considered an effective weapon in 4 out

of 10 cancers

no scalpel required

in 1976 aecl medical a division of

atomic energy of canada limited

developed a revolutionary double pass

accelerator which streamlined linear

accelerator designs by using

electromagnets to send beams through a

target twice instead of once

the therak-25 was one such double pass

machine 7 feet high and 12 feet wide

smaller than previous accelerators also

unlike the accelerators of old the

therac-25 was run principally by

software instead of hardware lines of

code instead of interdependent physical

mechanisms

in 1983 aecl performed a safety analysis

on the new machine and started selling

the therak25 to excited customers

this state-of-the-art device was in high

demand

however

left out of that 1983 analysis was any

interrogation of the software that ran

these complicated devices of the code

based on the older therak-20 model and

written by a single person

a coding hobbyist

who left the company in 1986.

he remains unidentified to this day

two weeks after katie yarborough told

her technician that she felt a burning

sensation during her cancer treatment

there was a red mark the size of a dime

on her chest

and directly opposite that mark a larger

disc on her back

tim still the medical physicist at

kennestone examined her

that looks like the exit dose made by an

electron beam

he said

it looked nothing like what could be

created by her prescribed 200 rad dose

the physicist later estimated what

actually hit yarborough was closer to 20

000 rats hundreds of times more than

what you'd receive standing inside a

failed reactor at fukushima daiichi

but dr still wasn't able to recreate a

beam of that strength with the machine

himself so he contacted a professional

organization to tell them what had

happened he quickly got a call from the

aecl in response telling him to stop

making these claims without any proof

they assured him that such an overdose

simply wasn't possible

over the next few weeks the dime-sized

red circle on yarborough's chest became

a hole

skin grafts failed as any new tissue

simply rotted away

her left breast recently cancer-free

had to be entirely removed

her left arm

was now immobile

many sources report it was though a

slow-motion gunshot wound had gone

through her chest and out of her back

yarburo would hire a lawyer and sue the

hospital and aecl in the october of 1985

but she wouldn't live to find out the

reason why nanoscopic bullets

had done this to her

seven weeks later concerningly similar

to katie yarbrough a 40 year old woman

with cervical cancer arrived for her

most recent therak-25 treatment at the

hamilton regional cancer center in

ontario canada

she too was hit with a slow-motion

bullet complaining of tingling electric

shocks during treatment

it would be later estimated that what

the therak operator had mistakenly

irradiated her hip width several times

was a total

of 17 000 rats

a larger dose than what harry dogly and

junior or louis slaughten received

from the demon core

the aecl was informed immediately and

had an engineer dispatch to examine the

unit

the micro switches that controlled the

position of the unit's turntable were

deemed faulty and a software change to

constantly check the turntable position

was introduced

aecl would later claim in a september

letter to customers that this change had

increased the safety of the therac-25 by

five orders of magnitude

but

the cervical cancer patient died a month

before this pronouncement

on november 3rd

her official cause of death was her

cervical cancer

though an autopsy revealed that if she

had lived her hip

obliterated by high energy radiation

would have to have been entirely

replaced

five days later a letter from the

canadian radiation protection bureau

begged aeco for hardware fail-safes and

additional software changes but nothing

came of it

a month after the heavily irradiated

cervical cancer patient died it happened

again

a therak-25 unit at the yakima valley

memorial hospital in washington state

supposed to be now 9 million percent

safer hit another cervical cancer

patient in the hip with more radiation

than what cecil kelly endured when a

whirlpool of plutonium

went prompt critical in his face

thankfully the woman ultimately suffered

only minor disability and scarring

the more impactful outcome was that

doctors and therak-25 operators in the

u.s and canada were now talking to each

other

something was going on that the aecl

obviously wasn't addressing or didn't

care to

two months later aecl declared that

after careful consideration we are of

the opinion that this damage could not

have been produced by a malfunction of

the therac-25 or by any operator error

end quote

however

over the next 12 months

therap-25 malfunction and operator error

would kill three cancer patients

the therac-25 software likely had around

100 000 lines of code

small by today's standards but

complicated nonetheless

and error-prone

operators would later testify that they

encountered as many as 4 serious error

messages a day

many of those errors would simply read

malfunction with a number from 1 to 64.

these numbers were not explained not by

aecl not in any

manual operators also admitted that they

became accustomed to this ambiguity

rather than fearful of it

they could and did simply press p to

proceed

without knowing whether or not an error

code was benign or potentially deadly

it was part of the job to keep an

expensive and sought after machine like

the therak running

malfunction 54 one of these mysterious

undefined errors would turn out to be

the one you couldn't skip past

but it would take three catastrophes

before anyone figured out what they were

allowing to happen

[Music]

voyn ray cox lay beneath the therak-25

unit at the east texas cancer center in

tyler texas for his ninth cancer

treatment a technician set his dose at

180 rads

then she noticed a mistake

she had selected x for x-ray instead of

e for electron beam

she quickly moved the cursor made the

change and activated the machine

malfunction 54.

used to this by now she hit proceed

anyway

mr cox then felt a powerful shock

according to reporting done by barbara

wade rose cox tried to get up but

because the intercom for the room just

happened to be broken that day the

technician didn't see him struggling

or hear him screaming

so she hit him again

another shock ripped through mr cox

the technician only stopped when she

heard cox slamming the door she was

behind with his fists

he was examined by physicians

and sent home

told to return if anything changed

a few weeks later he returned to the

hospital

spitting up blood

[Music]

after the accident no one can reproduce

malfunction 54.

aecl told the hospital that an overdose

was impossible suggesting maybe it was

indeed an electric shock that produced

the sensations mr cox felt but

ruled that out too

the company claimed it knew of no other

similar accidents so the use of the

therak unit resumed 17 days later

voin ray cox died the following august

after receiving a calculated dose higher

than the worst dose a liquidator would

receive when the chernobyl nuclear power

plant exploded

just a month later

only four days after the therac at the

east texas cancer center was back online

66 year old bus driver vernon kidd

walked through the lobby on the way to

his scheduled treatment

he was to have a therak-25 aimed at the

skin cancer on his face

malfunction 54

treatment proceeded

a loud noise brought the technician back

into the room to find mr kidd writhing

in pain

confused

he said it had felt like something hit

him on the side of the face

he saw a flash of light

heard an intense

sizzling sound

the therak-25 unit at the center was

shut down until the cause could be

determined

verdan kidd died a month later from

radiation-induced damage to his brain

and brain stem

his death

four months before the death of voyn ray

cox was the first recorded fatality from

radiation treatment

in medical history

the therac technician on site that day

and physicist dr fritz hager stayed the

weekend after the kid accident

attempting to recreate malfunction 54

the malfunction that aecl said wasn't

possible

they changed the machine's modes moved

the cursor quickly up and down and typed

in different treatment instructions for

hours upon hours and then suddenly

they did it

dr hager telephoned to aecl immediately

the fda already investigating the

accidents declared the therak-25

defective and demanded a corrective

action plan or cap from the company

a letter soon went out from the aecl to

all therac-25 users

quote effective immediately and until

further notice the key used for moving

the cursor back through the prescription

sequence must not be used for editing or

any other purpose

end quote it was something that everyone

had missed and it was finally going to

be fixed

but

even after everyone knew what

malfunction 54 was and how to fix it

the accidents continued

you don't think of software as something

being able to fail

once working code is in a computer how

could it bend like a steel beam or break

like a pane of glass

but like any machine there's a

difference between how it's supposed to

be used in theory and how it's actually

used in practice

the therac-25 used magnets to filter and

control powerful beams of radiation

magnets that after an input was received

physically took eight seconds to move

everything into position

fine in theory

what dr hager had figured out was that

if an operator set radiation levels and

then made a change to those levels

within the eight seconds it took for the

magnets to move

the change was not detected the magnets

were already in motion

this could and did allow powerful

unfiltered beams of radiation to strike

patients

this animation reproduced in spanish

shows the sequence of events needed to

produce malfunction 54.

in theory an operator would make a

change and then wait for the magnets in

the machine to move

but an experienced operator working with

the therac every single day encountering

multiple error messages a shift is in

practice more than likely of making a

change like changing a beam from x-ray

to electron within 8 seconds

there was no code in place to check

whether the prescribed input on the

monitor match what the machine was

actually set up to do

the therac-25's reliance on software and

not hardware interlocks like previous

models

also meant that input errors didn't have

mechanical fail-safes that wouldn't

listen to the mistakes that ended up

fatally irradiating people

with malfunction 54 finally identified

aecl sent the first corrective action

plan to the fda part of which were

changes to the therax software to tell

the machine where the cursor actually

was

the cap was revised twice by the fda

over the next few months and tharac25s

were back in use before the end of the

year

six weeks later

a therak unit killed again

on january 17th 1987 glenn dodd 65

walked into the yakima valley memorial

hospital for treatment of a carcinoma

his disease was to be flooded with 86

rats

the therak instead bombarded the man's

chest with ten thousand

he died from acute radiation poisoning

three months later

the staff at yakima reportedly stopped

using the machine altogether after this

accident they were paranoid they thought

this had been fixed

this was safe

what did they miss

what was going on

eventually it was discovered that the

therac25 had another invisible software

error

inside the code was a so-called

housekeeper task that would constantly

check whether or not the machine's

turntable was in the correct position

make adjustments if necessary and then

revert to zero

anything other than a zero in the code

therefore was an error and the machine

would not proceed with treatment again

good in theory

however like your car's odometer this

code ticked up checks only until a

certain value in this case one byte of

memory

256

after that it would tick over to zero

out of necessity

if a technician entered an erroneous

treatment at this precise moment

just an instant before the next check

the computer would read a zero

no errors

and proceed

this is called arithmetic overflow

and it's what killed glenn dodd

in february 1987 the fda again declared

the therak 25 defective recommended that

all units be taken out of service until

corrective action could be taken

finally the accident stopped

after months of revisions aecl told

customers that the fda had accepted a

final corrective action plan it included

23 software changes and six hardware

safety features

the largest of which was a dose per

pulse monitor affixed to the machine

that would shut down dangerous doses

even if all the software safety checks

failed

however

unsurprisingly

after high profile and unprecedented

accidents

the therak was no more

in 1988 aecl dissolved their medical

division and lawsuits from families were

settled out of court

at the time of this recording with a

cursory search i was unable to find any

hospital actively using a therap-25

even if they are which isn't unlikely

the machines are probably under a

different name or company

even without any further incidents to

point to today

no hospital wants to draw active

comparison

to disaster

today the therac25 is more a staple of

ethics and computer science class

required readings than it is of medicine

a unique case study of what can go wrong

when new technology is trusted

implicitly and when ethical decision

making malfunctions aecl assumed that

the software for the therac-25 written

by a single unidentified hobbyist and

imported from the older therak-20 model

did not have residual software errors to

be tested

it didn't consider how the machine was

being used in practice never envisioning

something like malfunction 54

the company repeatedly denied knowing of

any accidents and believed that

overdoses were impossible

it made proclamations like five orders

of magnitude increases in safety that

were physically impossible

today the fda requires documentation on

all software for new medical products

which the therac-25 didn't have

that can be investigated independently

ten years after the deadliest software

errors in history reporter barbara wade

rose asked bill bird the lawyer for the

first therac victim katie yarborough to

comment on the events

quote

the thing that amazes me

is that the people who develop these

machines are surely some of the most

brilliant people in the world

this machine was unbelievably

sophisticated

and yet

nobody would have gotten hurt

if somebody had just used common sense

until next time

[Music]

[Music]

[Music]

[Music]

you