DDoS Attack Analysis: Insights, Mitigation, and Future Preparedness

Executive Summary

Earlier this year a major website faced an aggressive Distributed Denial of Service (DDoS) attack, leading to an unprecedented traffic surge with nearly two billion requests within a few minutes. This report offers a comprehensive analysis of the incident, the response, and the lessons learned to enhance future preparedness.

Incident Overview

The attack began with an initial influx of suspect traffic that quickly escalated. A singular IP was observed pushing 5,000 requests per second, signaling the start of a large-scale DDoS attack involving numerous IPs. This rapid escalation overwhelmed the website’s infrastructure.

Infrastructure Response

The website’s defensive infrastructure, consisting of load balancers, Apache servers, Varnish and go caching, and a MySQL cluster, executed auto-scale and auto-recovery operations as designed. The architecture prioritized edge failure to protect the database from excessive load. Despite this, there were two instances where the database connections were maxed out, highlighting areas for improvement.

Detection and Cause Analysis

The attack was identified through application monitoring and subsequent Cloudflare logs. A detailed root cause analysis revealed:

1. Site Downtime Reason:
   The website was overwhelmed by the sheer volume of the DDoS attack.

2. Cloudflare’s Inadequate Blocking:
   The attackers used “clean” IPs that bypassed known rate-limiting protocols.

3. Ineffective Rate Limiting:
   The attack exposed the limitations of a per-IP rate limiting setup, suggesting the need for a more cumulative approach.

4. Load Balancer’s Traffic Capacity:
   The load balancer’s artificially low number of connections per second was intended to protect backend resources but proved insufficient against the attack’s scale.

5. Database Connection Errors:
   The database reached its maximum connection limit due to the high volume of traffic.

Mitigation and Resolution

Immediate adjustments were made to Cloudflare’s configurations to better handle the traffic surge. The infrastructure team increased resource allocations and blocked malicious IPs at the load balancer level. These measures mitigated the attack’s impact and restored site functionality.

Lessons and Future Strategy to Better Mitigate DDOS Attacks

Effective Responses:

• Quick reconfiguration of Cloudflare settings.
• Resilient infrastructure performance despite peak traffic loads.

DDOS Improvement Opportunities:

• System Enhancement: Implement additional layer 7 DDoS protection to complement Cloudflare’s capabilities.
• Process Optimization: Develop a dedicated DDoS alert system and improve communication protocols between incident management and cybersecurity teams.

DDOS Forward Looking Planning:

• API Refactoring: Ongoing development to employ websockets aims to reduce server load and ease cache clearance during high traffic periods.

Conclusion of how to better prepare for a DDOS

This incident highlights the necessity for continuous cybersecurity vigilance and adaptive strategies. The evolving scale and complexity of DDoS attacks require a proactive and collaborative defense approach. By learning from this event, we can better prepare for and mitigate future attacks, ensuring the resilience and availability of our digital infrastructure.