How I survived a DDoS attack

Summary

In this video, Jeff Gearling shares his experience of his website being hit by a massive DDoS attack, detailing the steps he took to recover it. After initially underestimating the risk, Jeff found himself facing a barrage of over 3,000 requests per second from a global botnet. Through trial and error, he implemented Cloudflare for protection, mitigating the attack. Jeff also reflects on lessons learned, including the importance of good monitoring and alerting systems, and the evolving nature of DDoS threats. He ties the event to the broader context of cyberwarfare, particularly in the ongoing conflict between Russia and Ukraine.

Takeaways

• 😀 A DDoS attack targeted the website, initially thought to be a standard DoS (Denial of Service) attack.
• 😀 The DDoS attack involved thousands of computers worldwide, generating over 3,000 requests per second, overwhelming the server.
• 😀 Initially, blocking IPs and rate-limiting requests in Nginx didn’t work due to the volume and complexity of the attack.
• 😀 Switching the website’s traffic through Cloudflare helped mitigate the attack by blocking millions of malicious requests.
• 😀 The attacker’s traffic came from multiple countries, including Indonesia, Russia, Brazil, China, and more.
• 😀 Cloudflare handled over 25 million requests, protecting the website from further damage and providing necessary DDoS protection.
• 😀 The attack had severe consequences on the website's backend infrastructure, but switching to Cloudflare significantly stabilized the site.
• 😀 The lesson learned was the importance of monitoring and alerting to detect and respond quickly to DDoS attacks.
• 😀 The experience highlighted the risk of relying on home servers and public IPs for services exposed to the internet.
• 😀 The ongoing cybersecurity challenges, especially regarding DDoS attacks, have connections to global events, like the Ukraine-Russia conflict, which involves large-scale DDoS warfare.
• 😀 In the end, the website owner adapted by using Cloudflare permanently, accepting that external services are essential for robust online defense.

Q & A

What is the main difference between a DDoS and a DoS attack?

A DoS (Denial of Service) attack involves a single machine sending an overwhelming amount of traffic to a server, causing it to crash or become unavailable. A DDoS (Distributed Denial of Service) attack, however, involves multiple machines from around the world (often part of botnets) flooding the target with traffic, making it much more difficult to stop.

Why did the website get attacked after the video was posted?

After the video was posted, the website's visibility increased when it was shared on Hacker News, leading to a surge in traffic. The attacker likely targeted the site when it gained significant exposure, exploiting the increased traffic to launch a DDoS attack.

How did Jeff identify that the attack was a DDoS and not a regular traffic spike?

Jeff identified it as a DDoS attack by reviewing server logs and noticing that requests were coming from thousands of different IP addresses across the world, specifically from countries like Indonesia, Russia, and Brazil. Additionally, the traffic was abnormally high and consisted of post requests that the server couldn't process.

What steps did Jeff initially take to mitigate the DDoS attack?

Jeff first attempted to block the attacking IPs using Nginx configuration. He also tried rate limiting requests, but this caused issues for legitimate users. He then decided to put the website behind Cloudflare to help mitigate the DDoS traffic.

What role did Cloudflare play in handling the DDoS attack?

Cloudflare acted as a shield for the website, absorbing and filtering the massive amount of traffic. By routing all traffic through Cloudflare's network, Jeff could block harmful traffic while keeping legitimate users able to access the site. Cloudflare blocked millions of requests and allowed the site to become stable again.

What mistake did Jeff make when setting up Cloudflare, and how did he fix it?

Initially, Jeff blocked IPs in Nginx to protect his VPS, but that inadvertently blocked Cloudflare's own IP addresses, causing issues. He fixed this by disabling the IP blocking in Nginx and setting up a firewall rule on DigitalOcean to only allow Cloudflare servers to access his VPS.

What specific challenges did Jeff face after switching to Cloudflare?

After switching to Cloudflare, Jeff encountered issues with his complex Drupal setup. His site’s subdomains were offline, and he couldn't save new blog posts due to conflicts between Cloudflare’s proxy and Drupal’s backend security. He had to adjust settings to allow for proper functioning.

What did Jeff learn about risk management during this DDoS attack?

Jeff learned that he needed to adjust his appetite for risk. While he had been running services from his home lab without much concern, he now realized that relying solely on public IP addresses and self-hosted services left him vulnerable to DDoS attacks. Using third-party services like Cloudflare has become necessary for maintaining site availability under high traffic volumes.

How did the DDoS attacks relate to the ongoing conflict between Ukraine and Russia?

The DDoS attacks are part of a broader cyberwar that has escalated due to Russia's invasion of Ukraine. Many individuals and groups around the world have launched DDoS attacks against Russian websites in response to the invasion, with Ukrainian cyber defense initiatives playing a key role.

What does the term 'botnet' mean, and how does it relate to the DDoS attack Jeff faced?

A botnet is a network of computers or devices that have been infected and are controlled by an attacker to perform malicious activities, such as sending large volumes of traffic in a DDoS attack. Jeff’s website was targeted by one such botnet, which used thousands of compromised devices worldwide to overwhelm his server.