03 dec

EvoSwitch Hall 1/2 problems [Update]

Time: December 3rd 2009 02:05:00(CEST)
Expected Downtime: Unknown
Location: EvoSwitch
Reason: under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 02:20] All services are back online again, we are currently investigating the cause if this issue.

[UPDATE 02:58] It seems some feeds had a short power dip. This is not certain yet. We are still investigating.

[UPDATE 03:12] Some colocation customers are causing ethernet loops, these interfaces are being shut now.

[UPDATE 03:17] to clarify, the loops did not start the problems, it seems that the colocation customers switches are causing loops after the power dip.
All looping interfaces have been shutdown.

[UPDATE 03:39] Four extra engineers are at the datacenter. And some more underway. If your server is still not online please contact support via email.

[UPDATE 07:20] A customer in Hall 2 who was replacing network equipment caused a loop which caused partial unreachability in Hall2.

[UPDATE 07:40] The customer who caused the loop has been taken offline and all services have been restored.