CleanMPG knocked out for most of Friday, January 1, 2016

xcel · Jan 2, 2016

Hi All:

The day we shut down the old hosting service and server in Dallas, the new one CleanMPG is built on in Atlanta was hit hard...

Here is that story.

Connectivity Issues - Atlanta

Update - We have received word that all of our Atlanta IP space is now being announced. We are still fine tuning our geographic blocking, so people in the affected regions may see intermittent loss of connectivity over the next hour or two.
Jan 2, 05:24 UTC

Update - Zayo has begun bringing our IP ranges online.

For the short term, we will be using BGP communities to attempt to block Asia Pacific, the Middle East, South America, and others, hopefully leaving us only with traffic from North America and Western Europe. Blocking geographic regions this way is the only way to make sure that large botnets won't be able to launch further attacks.

We hope to have all of our Atlanta IP space advertised in the next several hours.

Alex Forster
Network Engineer, Linode
Jan 2, 04:42 UTC

Update - Our upstream provider has informed us that they plan to restore network connectivity to our systems gradually over an undisclosed period of time. There is no ETA at the moment, but our engineers and leadership team members are working closely with theirs to accelerate this process as much as possible. We will provide another update as soon as more information becomes available.
Jan 2, 02:36 UTC

Update - Our upstream provider reports that they've made some progress on mitigating the numerous DoS on their infrastructure, but due to the varying nature of the target, scope, and size of the attacks, network connectivity has not yet been restored. We are working closely with their team to remain informed on their progress, and will update this page as soon as there are any further developments. Once again, we sincerely thank you all for your patience.
Jan 1, 23:02 UTC

Update - The DoS attack in Atlanta is ongoing, and is still causing major network disruptions in our Atlanta location. We are continuing to work with our upstream provider to restore network functionality as quickly as possible. Thank you again for your patience.
Jan 1, 20:22 UTC

Update - The attack in Atlanta is still ongoing and causing significant connectivity issues. We're closely working with our upstream provider to ensure that connectivity is restored as soon as possible. We apologize for any inconvenience these issues are causing and we thank you for your patience.
Jan 1, 17:40 UTC

Identified - We're currently engaged with our upstream provider in mitigating a large, distributed DoS attack targeting our infrastructure in Atlanta. Users may experience packet loss and issues with connectivity as we work to resolve this. We thank you for your patience.
Jan 1, 15:29 UTC
Click to expand...

Wayne

ALS · Jan 2, 2016

Hi Wayne,

I figured that something was up when I couldn't connect to the board.
Glad the software is fixed and running smoothly.

Al

xcel · Jan 2, 2016

Hi Al:

Yeah this one was the Hosting group this time around. They are still trying to clean up the mess as they just brought their servers adn services online whihc in turn brought us online.

Hopefully for good now?

Wayne

xcel · Jan 3, 2016

Hi All:

It may finally be over and what a mess. Sorry for the inconvenience on both Friday and Saturday as well.

Connectivity Issues - Atlanta

Monitoring - We believe that we have closed all of the attack vectors that can lead to a DDoS taking down the entire Atlanta datacenter. We are monitoring to confirm this.
Jan 3, 15:24 UTC

Update - The Atlanta data center is still experiencing network connectivity issues due to the DDoS attack. Our Network Engineers have been working throughout the night to restore connectivity.
Jan 3, 12:17 UTC

Update - We are still in the process of mitigating the DoS attack in our Atlanta data center. Our networking team is working with our upstream provider to restore normal connectivity to the data center.
Jan 3, 07:43 UTC

Update - Our Networking team is still actively engaged in restoring network connectivity. We'll continue to keep this status post updated with the latest information.
Jan 3, 04:11 UTC

Update - There has been some progress in restoring network connectivity to our Atlanta location, but our engineers are still hard at work to fully restore functionality to this datacenter. Thank you very much for your patience.
Jan 3, 02:06 UTC

Update - We're still actively working with our upstream provider to get the transit link hardened and getting connectivity restored. We will continue to keep providing updates until the issue is resolved.
Jan 3, 01:03 UTC

Update - The process of hardening our transit link and restoring connectivity in Atlanta is ongoing. We'll continue to provide updates as we work to resolve this issue.
Jan 2, 23:01 UTC

Wayne

Log in or Sign up

CleanMPG knocked out for most of Friday, January 1, 2016

xcel PZEV, there's nothing like it :) Staff Member

ALS Super Moderator Staff Member

xcel PZEV, there's nothing like it :) Staff Member

xcel PZEV, there's nothing like it :) Staff Member

Share This Page

Home

Media

Members

Useful Searches