Time: September 25th 2008 16:30:00
Leaseweb Update
Dear customer,
as you might or might not have noticed, there were some network issues at Leaseweb on Wednesday, September 24. Unfortunately, this affected our phone system as well, so for a period of time you could not reach our support.
We apologize sincerely for any inconvenience you might have experienced from this.
Currently we are still investigating the causes in detail. With this newsletter we would like to inform you about the current state of these investigations:
Starting 13:20 CET and continuing through the afternoon, we experienced some severe DDoS attacks on several IPs. The heavy load continued even after null routing the affected IPs. This was caused by multiple blade failures in one of our datacenters in combination with software errors.
At around 16:00 CET the network was stable again but got under a new stress attack right away. The connection between EVO and other parts of our network still experienced some package losses. We could keep the effect mostly local, so the overall network remained stable. At 19:00 everything was finally totally contained.
In summary, we can only say for sure at this stage that the instability was due to some combination of the DDoS attacks and hardware and software issues which were most likely caused or at least accelerated by the DDoS.
As the problems fluctuated and we took countermeasures right away, not all servers were affected and even the ones that were mostly only experienced very limited outages.
Nevertheless and again, we apologize for the trouble you have had with connectivity, your servers, websites and customers.
Please note that we will update the software on Ring 3 tonight as a precautionary measure. (This will also be announced separately on the NOC site.)
For the future, we already started taking precautions for similar issues several weeks ago: we have been working on redesigning our network setup for two new Cisco CRS-1 Backbone Core routers. This top notch technology will reduce the potential points of failure and enable us to react even faster to network issues. Furthermore, we have been redesigning our telephone system with redundancy capacities. It will go online in a few days, so you will be able to reach us in such cases of emergency and we can deliver the usual good Leaseweb service.
On this as well as the current issues we will keep you updated. Especially for the latter, please also check for news on our NOC (Network Operations Center) site at http://noc.leaseweb.com/ where we always announce important network issues.
We would also like to remind you on this occasion on the ways to contact our support. For any issues, please feel free to contact us at +31 20 3162880 via phone or [email protected] via email. Please do so especially if you are still experiencing problems with relation to Wednesday???s issues.
Best regards
The Leaseweb B.V. Management