Serverless to Homeless - Case study

Summary

The video discusses a case where a user received a $100,000 bill from Netlify for a static website, highlighting the issue of unexpected costs with cloud service providers. It explores the possibility of a Distributed Denial of Service (DDoS) attack and the lack of automatic DDoS protection, comparing bandwidth costs across different providers. The video emphasizes the importance of being aware of service tiers and potential expenses, and ends with Netlify waiving the charges after public discussion, raising questions about trust and transparency in cloud platforms.

Takeaways

• 📈 A user received a $100,000 bill from Netlify for a static website, highlighting a potential issue with cost management and unexpected expenses.
• 🚀 Static websites are typically expected to have minimal costs, with some providers offering pro plans that include unlimited bandwidth within certain tiers.
• 💸 Overshooting the included resources in a plan, such as serverless compute or bandwidth, can lead to significant charges based on the provider's rates.
• 🌐 The incident at Netlify (nlii) involved a single day with 60 terabytes of bandwidth usage, which is highly unusual and indicative of a possible DDoS attack.
• 💰 Comparing costs, Netlify charges $55 for 100 GB of bandwidth, which is significantly more expensive than cloud providers like Hetzner or AWS.
• 🛡️ Lack of automatic DDoS protection can be a vulnerability; however, some providers offer emergency DDoS modes to help mitigate such attacks.
• 🔧 The user's website was affected by a DDoS attack focused on a single file, possibly an audio clip, leading to massive data transfer.
• 🌐 Netlify's response suggested hosting music on third-party platforms to reduce bandwidth usage, which may not be ideal for all users needing to host certain assets.
• 🤝 Netlify CEO eventually responded to the situation on Hacker News, stating that the user would not be charged for the excessive bandwidth usage.
• 📉 The incident raised concerns about the trustworthiness of platform providers and the potential for being unknowingly targeted by DDoS attacks.

Q & A

What was the initial bill amount that the user received from Netlify?

The user initially received a bill for almost $104,000 in USD from Netlify.

What type of website incurred such a high bill?

The high bill was for a simple static website, which typically should have minimal to no hosting costs.

How did the user initially react to the bill?

The user initially thought it was a joke or a scam, but after checking their dashboard, they realized it was a legitimate overdue bill.

What is the usual cost for a pro plan on platforms like Netlify or Vercel?

The usual cost for a pro plan on platforms like Netlify or Vercel is around $20 a month, with an uninterrupted tier included.

What caused the user to exceed the plan's bandwidth limit?

The user was charged for exceeding the bandwidth limit due to a spike in traffic, with 60 terabytes of bandwidth used in a single day.

How does the cost of bandwidth on SaaS platforms like Netlify compare to cloud providers?

SaaS platforms like Netlify charge significantly more for bandwidth compared to cloud providers. For example, Netlify charges $55 for 100 GB, while cloud providers like Hetzner and AWS have much lower rates or even offer free tiers for certain amounts of traffic.

What is the issue with automatic DDoS protection on these platforms?

Automatic DDoS protection is difficult to implement because it's challenging to differentiate between legitimate and malicious traffic during a distributed denial-of-service attack.

What did Netlify suggest as a solution to prevent such high costs in the future?

Netlify suggested hosting music on third-party platforms like YouTube, Bandcamp, or SoundCloud to reduce bandwidth usage, regardless of the site's popularity.

How did the user feel about Netlify's response to the situation?

The user disagreed with Netlify's response, feeling that the platform was placing blame on the user and not providing adequate solutions for such incidents.

What was the final outcome for the user who received the $100,000 bill?

Netlify's CEO responded on Hacker News, stating that 100% of the charges were removed and that their policy is not to shut down free sites during traffic spikes that don't match attack patterns.

What concerns do users have about the transparency and fairness of billing practices on these platforms?

Users are concerned that they cannot verify the legitimacy of the charges or protect themselves from potential fraudulent activities by the platform, as the databases and billing systems are controlled by the platform companies.

Outlines

00:00

💸 Unexpected $100,000 Bill from Netlify

The video begins with the startling revelation that the user received an invoice for $100,000 from Netlify for a simple static website. The speaker discusses the potential reasons for such a high bill, emphasizing the importance of understanding one's hosting provider's pricing structure, especially for services like serverless compute, bandwidth, and image optimization. The speaker shares a personal experience with Netlify, detailing how they were mistakenly billed for an excessive amount due to a spike in bandwidth usage, suspected to be a DDoS attack. The discussion highlights the high costs of bandwidth through SaaS providers like Netlify and WorldPress compared to traditional cloud providers, and the lack of automatic DDoS protection, which leaves users vulnerable to unexpected expenses.

05:01

🚫 Netlify's Response and Recommendations

The speaker criticizes Netlify's response to the situation, which suggested that the user's static assets, such as sound files, should be hosted on a third-party platform to reduce bandwidth usage. The speaker argues that these assets are necessary for the website's functionality and should be delivered through the edge network provided by Netlify. The video also addresses the issue of DDoS protection and the difficulty for providers to automatically detect and prevent such attacks. The speaker shares their own experience of reducing the bill by attributing 20% of the cost to the DDoS attack and further negotiating with Netlify, which ultimately led to the charges being waived. The speaker expresses concern about the lack of transparency and control users have over their data and the potential for misuse by platform companies.

10:01

📢 Conclusion and Future Considerations

In the concluding part of the video, the speaker reflects on the incident and its resolution, noting that the user was not charged for the excessive bandwidth usage. The speaker commends Netlify's policy of not shutting down free sites during traffic spikes that don't match attack patterns, but also acknowledges the stress and potential financial burden such a mistake can cause for users. The speaker raises questions about the trustworthiness of platform providers and the lack of recourse for users in such situations. The video ends with an invitation for viewers to share their thoughts in the comments and a prompt to like and subscribe for more content.

Keywords

💡bill

In the context of the video, a 'bill' refers to an invoice or statement of charges that a service provider sends to a customer for goods or services rendered. The video discusses an unusually high bill of $100,000 for a static website, which is typically expected to have minimal costs.

💡static website

A 'static website' is a type of website whose content is fixed and does not change or require frequent updates. It is typically composed of HTML, CSS, and JavaScript files that are served directly to the user without the need for a backend database or server-side scripting.

💡serverless compute

Serverless compute is a cloud computing execution model in which the cloud provider dynamically manages the allocation and provisioning of servers. With serverless, developers can build and run applications without having to explicitly provision or manage servers.

💡bandwidth

In the context of the video, 'bandwidth' refers to the data transfer capacity of a network connection. It is the rate at which data can be transferred over a communication link. High bandwidth usage can lead to significant costs, as seen in the $100,000 bill incident.

💡DDoS attack

A Distributed Denial of Service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted server, service, or network by overwhelming it with a flood of internet traffic. This can result in the target being unavailable to legitimate users.

💡Cloudflare

Cloudflare is a web performance and security company that provides services such as DDoS mitigation, web content acceleration, and security features to help protect websites from online threats.

💡CDN (Content Delivery Network)

A Content Delivery Network (CDN) is a distributed network of proxy servers and data centers that deliver content to users from the server closest to their geographic location. This improves website performance and reduces latency by bringing content physically closer to users.

💡Netlify

Netlify is a cloud computing platform that provides a suite of services for web developers, including continuous deployment, hosting, and serverless functions. It is known for its support of static websites and Jamstack architecture.

💡AWS (Amazon Web Services)

Amazon Web Services (AWS) is a comprehensive cloud computing platform provided by Amazon that offers a wide range of services such as computing power, database storage, content delivery, and other functionalities to help businesses scale and grow.

💡cost reduction

Cost reduction refers to the process of decreasing expenses associated with the production of goods or services. In the context of the video, it relates to strategies for reducing the costs of hosting a website, such as using third-party platforms or optimizing content delivery.

💡support response

A 'support response' is the reply or assistance provided by a customer service team to address a customer's issue or inquiry. In the video, the support response from Netlify is discussed in relation to the handling of the $100,000 bill situation.

Highlights

A user received a $100,000 bill from Netlify for a simple static website, raising concerns about cost management and prevention strategies.

The expected cost for hosting a static website is minimal, with most plans offering an uninterrupted tier to cover usage within limits.

Exceeding the plan's limits, such as serverless compute or bandwidth, results in additional charges based on the provider's rates.

A similar incident occurred at NLII with a user receiving an unexpectedly high bill, indicating a potential issue with cost management.

The user's bill was almost $104,000 in USD, which was initially dismissed as a joke or scam but was confirmed upon checking the dashboard.

The peak day of February 16th showed a usage of 60 terabytes of bandwidth, which is unusually high and indicative of a possible DDoS attack.

Bandwidth costs from SaaS providers like World and Netlify are significantly higher compared to cloud providers like Hetzner and AWS.

Cloudflare offers zero cost for data transfer out of the internet with their CDN service, contrasting with the high charges from other providers.

The lack of automatic DDoS protection is noted, as implementing such protection is challenging due to the difficulty in distinguishing between legitimate and malicious traffic.

Providers like Cloudflare offer emergency DDoS mode to absorb the attack and allow legitimate users through after solving a challenge.

The high bandwidth usage was attributed to ancient user agents and Google Cloud addresses, suggesting a possible DDoS attack.

Netlify's response to the incident included a recommendation to host music on third-party platforms to reduce bandwidth usage.

The user's dissatisfaction with Netlify's response highlights the issue of placing blame on the user for necessary static assets.

The incident raises concerns about the trustworthiness of platform companies and their internal systems, which are black boxes to consumers.

Netlify CEO's response on Hacker News assured that the user would not be charged for the bill, and all charges were removed.

The policy of not shutting down free sites during traffic spikes that don't match attack patterns was emphasized, but apologies were made for the lack of clarity in initial support replies.

The user's experience with the unexpected bill and the subsequent resolution process was shared, highlighting the importance of transparent communication from service providers.

Transcripts

so net lii sent a user a bill of

$100,000 for a simple static website

let's take a look at what this is what

went wrong and how you if you're using a

provider like worel or nettia or

something like that you can prevent this

Behavior or you can prevent this cost so

$100,000 is of course not a small amount

of course if you're hosting something

you know you're expecting if it is

static you're expecting it to be pretty

much zero cost at most at you know you

are probably on a pro plan on what sale

or even netlify which cost like $20 a

month but all of these plans even the

Pro Plan comes with an uninterrupted

tier right so what happens is that if

you are using the things within that

tier it'll just be included in your plan

but if you overshoot let's say for

serverless compute or bandwidth or image

optimization or anything like that then

they will just charge you based on you

know their rates so a similar incident

incident happened at nlii where somebody

received a $100,000 bill and uh it's to

say the very least it's a bit strange so

let's take a look at what happened so I

received an email from netlify last

weekend saying that I have

104,000 almost in USD in Bill overdue at

first I thought this is a joke or some

scam but after checking my dashboard it

seems like I'm truly owning them that

amount so I was like and think okay

maybe I got dos since netlify charges

$55 for 100 GB for the exceeding

bandwidth the peak day of February 16th

had this much amount which is like you

know 60 terabytes of bandwith in a

single day see so that's that's the

thing that is the thing that two things

wrong here the first one is that you

know this cost is outrageous if you take

a look at actual cost for bandwidth it

is not as expensive as $55 for 100 GB

right now why worel and why netlify and

why you know all these SAS providers

sell the bandwidth extremely expensive

worel is cheaper than this worel sells

it for 40 USD which I'm assuming like

they are reducing now I have heard some

things that Al is in talks with a lot of

people to reduce this number but let's

see what it is but netlify charges

$55 for 100 GB of bandwidth this cost is

insane right so if you take a look at

herzner for example see so with herzner

20 terab of traffic is already included

and extra traffic is € per month per TV

right so you can see that the markup

here is 55 times more 55 times is you

know its orders of magnitude more than

what a cloud provider like herzner is

costing you and even AWS doesn't charge

this month right AWS also has huge

charges and but it doesn't charge that

plus if you look at providers like Cloud

flare Cloud flare has basically zero

cost for this so Cloud flare basically

doesn't charge you anything at all if

you are transferring data out of

Internet with their CDN service or you

know with R2 I think as well which is

their S3 alternative so this number in

itself is bad but the second second

thing is that there is no automatic dos

protection which is which I agree like

implementing a Dos protection is by its

very definition extremely hard because

dos means that it's distributed denial

of service attack that means that all

across the world the computers are

pinging your IP address and your website

address and it's extremely hard to

determine that this particular visit is

a bot visit or is some malicious visit

and that particular visit is a legit

visit so what happens generally in cases

of Dos is that you anyway have to like

you know put a capture on every single

page of the website for every single

user at least for that time being so if

you go to Cloud flare and these

providers A lot of these providers

actually what they do is they offer you

a you know an emergency dos mode so when

you toggle that on everyone gets a

challenge page on top of there you know

whenever somebody's visiting your

website they'll see a challenge page

from that provider which itself absorbs

all the DS attack and then just lets

legitimate users pass through because

they have to solve a capture or

something like that so see this is what

what happened like it was dsed on a

single file which is you know a sound

file and it got a lot of terabytes in

data transfer and what they told what

netlify told is that told them that

after looking into this further it seems

like a lot of bandwith usage came from

some user agents that are quite ancient

and uses Google Cloud addresses this

would include devices such as this this

this so either you have a fan base with

a passion for older technology or this

was likely a Dos attack I mean somebody

who's getting 16 4 terab I'm assuming

that they don't have a fan base of that

particular audio clip as aggressive as

that so so this seems like a this seems

like a weird message weird email to

write to someone who is already stressed

about you know paying a $100,000 bill

but okay going forward I would recommend

hosting music on a thirdparty music

platform such as YouTube band camp or

SoundCloud and reduce your bandwidth

usage no matter how popular your site

becomes so see this is where I really

disagree with the with how netlify has

responded in this email because if I'm

hosting a website it is my website and I

need those static assets whether that's

an image whether that's a sound file

whether that's a vasm binary I need that

on the edge Network which you are

providing right that that was the whole

proposition that we will deliver files

very fast to you sure I can host my

files on S3 but the only reason I would

do that is to save money save cost not

from a point of view that somebody just

downloads my file 10 times and you know

I'll be just bankrupt I will be homeless

so I think this is slightly bad take

from netlify where it's it's putting the

blame on the user itself and especially

in the case where Theos attack happened

right so it's not like they're

complaining out of thin air that you

know something like that happened it's

especially bad it puts a bad image that

you can't rust netlify you know the next

time you're probably hosting an image

which is like 1 MB 2 MB you know you're

not running any Optimizer or let even go

of that let's just assume you know

you're hosting something like a Monaco

editor or a VSS code sort of instance

which we also do on on code Dam so if I

show you for example this instance over

here which let's say if you boot up this

playground so you're going to see that a

vs code like editor opens up now this

syntax highlighting which you see this

syntax highlighting can you see that

these keywords are of different colors

just like how it works in VSS code you

know how that is possible that is

possible through a vasam bilary called

onm so o GM if you Google this this

onm binary it's a web assembly Port of

something that is required for this

tokenization and colorful syntax which

you see so basically this vam binary or

this package is required for VSS code to

provide syntax highlighting and for

reasons which I don't want to get into

this video but you need to host this

binary under your own main domain right

so you have to have that as a static

asset on your domain it's also bundled

internally by a node you know it's

bundled by the build pipeline itself

right so how let's say if something like

this happens to code Dam or something

like this happens to a website which is

using solution like this then how would

you would you really say that you know

you can just pick your binary and just

host it somewhere else I mean that's not

the solution right we have to develop

better Solutions than just blaming it on

the user so we normally discount these

kinds of attacks to about 20% of the

cost which would make your new bill

$20,000 I've currently reduced it to

about 5% which is $5,000 I know this is

still a lot of money and I apologize for

the inconvenience I mean this is is

somebody who's imagine it from a

scenario like you are not sitting in US

you're sitting in a country like India

where 5,000 USD is probably months of

your salary or months of your savings

right for an average developer or for an

average person who's by the way like

earning relatively well in India also

this this sort of behavior is something

which is you know which which can be

heart crushing it can be very brutal for

somebody who's trying to just host a

website a static website and they're

just experimenting seeing things so yeah

it's it's actually bad so he also posted

this on Hacker News and nlii CEO

response our support team has reached

out to the user from thread to let them

know that they are not getting charged

for this so of course like 100% of the

charges charges are removed it is

currently our policy to not shut down

free sides during traffic Spike that

doesn't match attack patterns but

instead for giving any bills from

legitimate mistakes after the fact

apologies that this didn't come through

in the initial supporter reply so see

here people have also did done some

pretty wild Acquisitions on nlii that

how on Earth can I as a consumer be sure

that nlii has not paid somebody to DS me

I mean this is completely fair question

I would say but these are some of the

questions where you can't do anything

about it right so you can't do anything

at all right because these are the

platforms they are controlled by these

platform companies right so at the end

of the day their databases are black

boxes their databases and what they

report to you fundamentally are black

boxes what we can just assume is that

everyone has the best interest for

customers in their minds and nobody's

like messing around or you know just

tweaking numbers or you know doing

things like these to charge or

overcharge customers that's the best we

can think about just to answer this

question there is no way you in in the

world can tell that if netlify did that

or did not do that because it's their

own systems at the end of the day you

can't tell it from outside like watching

it watching this video like this there

is nothing I can do to tell like what

happened so yeah that's basically it

about this one I think it ended well

with the person at least not paying

anything I don't see like there is any

sort of resolution so far on this that

what exactly happened like who dsed so

that's it for this video I think it's at

least the end was well for at least the

person who got dsed I don't think there

is still any update on like they say

they have the support hasn't come back

with the IP information so that's still

work in progress yeah what do you think

about this let me know in the comments

below make sure you like And subscribe

thank you so much for watching and I'll

see you in the next video really

soon