Houston I have a problem

RFO: Partial DDoS of PointDNS services May 25-26th 2016

Follow

TIMELINE: 14:36 UTC 05/25/2016 to 07:00 UTC 05/26/2016

 

SERVICES AFFECTED:

  • dns6.pointhq.com

  • dns8.pointhq.com

  • dns9.pointhq.com

  • dns10.pointhq.com

  • dns11.pointhq.com

 

TIMELINE OF ACTIONS:

  • 2016-05-25 14:36 UTC - 5 of our DNS servers started to receive high amounts of traffic which started to saturate the network interfaces

  • 2016-05-25 14:41 UTC - We started investigating the issue along with traffic source assessment

  • 2016-05-25 14:50 UTC - We started rerouting traffic of the attacked DNS's to other DNS's which highlighted the target points

  • 2016-05-25 15:38 UTC - We identified the pattern of DDoS attack which was using domain registered in China and was patterned as malicious behaviour most likely using a botnet with more than 1k unique IP address from across the globe

  • 2016-05-25 16:31 UTC - Due to traffic volumes, we created firewall rules to filter-out attackers however, while this rule could be applied on our DNS's, and effectively drop all responses for attacker's requests, the incoming traffic was still able to saturate our interfaces due to a domain registrars unvalidated DNS point.

  • 2016-05-25 17:00 UTC - We have reported this problem to the reference domain registrar where the offending domain was registered, as well as attempting to directly contact the domain owner while we continued trying to manage the inbound traffic with rate-limiting on requests.

  • 2016-05-25 20:59 UTC - Working with our upstream partners we blocked the attackers and moved two dns servers (dns6.pointhq.com and dns8.pointhq.com) to completely new IP addresses once we had stabilised the interfaces. We have updated all records in PointDNS which were pointing to to the target IP addresses to match the new ones. At this point the impact on systems had been minimised, with service monitoring

  • 2016-05-26 07:00 UTC - Post attack, following monitoring of continued traffic filtering, we have begun the process of reverting all routing/IP changes to their original form

  • 2016-05-26 08:39 UTC - dns8.pointhq.com IP reverted to original: 23.236.59.22

  • 2016-05-26 11:41 UTC - dns9.pointhq.com IP reverted to original: 173.192.101.194

  • 2016-05-26 12:10 UTC - dns10.pointhq.com IP reverted to original: 81.95.158.50

  • 2016-05-26 14:24 UTC - dns6.pointhq.com IP reverted to original: 91.109.245.139

 

DISRUPTION SUMMARY:

The following is a summary of actual downtime on each DNS server over the course of the period of the DDoS attack from 14:36 UTC 05/25/2016 to 07:00 UTC 05/26/2016. The times indicated represent a sum total of availability impact for the duration, and do not represent a continuous period.

dns9.pointhq.com 0h 49m  
dns10.pointhq.com 0h 42m  
dns6.pointhq.com 0h 27m  
dns8.pointhq.com 0h 19m  
dns11.pointhq.com 0h 16m  
dns14.pointhq.com 0h 12m  
dns5.pointhq.com 0h 5m  
dns7.pointhq.com 0h 5m  
dns1.pointhq.com 0h 5m  
dns15.pointhq.com 0h 4m  
dns2.pointhq.com 0h 4m  
dns12.pointhq.com 0h 3m  
dns4.pointhq.com 0h 2m  
dns3.pointhq.com 0h 2m  

 

COMMENTARY:

While this attack was sizeable (the attack took out one of our geo-partners entire infrastructure in that region) and having multiple pan-global sources, the impact to customers was generally low with little effect on service usage levels for majority of PointDNS customers.

However, for a limited number of users of the service exclusively using the affected DNS servers, the impact was greater. And for some of those affected, the issue was being compounded due to some using the IP addresses of the DNS servers instead of the canonical names.

During this period, our response times to tickets was far from ideal, which was also further exacerbated by our status page updates not reflecting the progression of the incident, which was down to human error and communication oversight in an attempt to resolve the issue quicker. This had a knock-on effect of causing customers to be left without appropriate updates to progress to allow them to take decisions to minimise impact on them. Following this, we have added an internal review for DDoS incident management to guarantee better communication levels.

Was this article helpful?
0 out of 0 found this helpful

Comments