03 sep

[20100911] Peering connectio maintenance (Spain)

Startdate: September 11th 2010 23:00:00(CEST)
Enddate: September 12th 2010 05:00:00(CEST)

Dear customer,



Our supplier of our peering connection to Spain has announced a maintenance on their equipment. During this maintenance we will reroute traffic to different links.



No customer impact is expected, as we will reroute the excess traffic to different links, however during changing the networks paths and routing tuning you may see short latency increase or small packet loss.



The time zone is CEST (UTC+1).



LeaseWeb – NOC

28 aug

Easynet problems [Update]

Time: August 28th 2010 20:31:00(CEST)
Expected Downtime: Unknown
Location: Easynet
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 21:45]
There was a DDoS targeted at the router.
This attack was filtered a couple of minutes ago.

28 aug

Core router software update (CRS-EVO)

Startdate: September 1st 2010 06:00:00(CEST)
Enddate: September 1st 2010 07:00:00(CEST)

Dear Customer,

The outage our network experienced Friday 27-08-2010 (http://noc.leaseweb.com/status.php?i=557) was caused by a bug in the IOS-XR software which runs on our Cisco CRS-1 core routers.

More information about the bug can be read on Cisco’s page concerning this issue.

During this maintenance window we will install the update on our CRS-1 router located in EvoSwitch.
The router will be rebooted.
Before the maintenance starts we will reroute all traffic to the core router located in Telecity-2.
Customer traffic loss is not expected.

Kind regards,

LeaseWeb NOC

27 aug

Major network issue [Update]

Time: August 27th 2010 10:09:00(CEST)
Expected Downtime: Unknown
Location: All
Affected IP ranges: All
Reason: Under investigation
*******************************

Dear Customer,

At the moment we are experiencing major network issues, both core routers are dropping BGP sessions.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.


Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 11:25] The situation seems stable again, we are still investigating.

[UPDATE 14:16] Our core routers remain stable, we are working on a RFO.
The RFO will be ready beginning next week.
Upon request your account manager will send you the RFO.

27 aug

Major network issue [Update]

Time: August 27th 2010 10:09:00(CEST)
Expected Downtime: Unknown
Location: All
Affected IP ranges: All
Reason: Under investigation
*******************************

Dear Customer,

At the moment we are experiencing major network issues, both core routers are dropping BGP sessions.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.


Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 11:25] The situation seems stable again, we are still investigating.

27 aug

Network interuption at evoswitch

Time: August 27th 2010 09:44:00(CEST)
Expected Downtime: 1 minute
Location: Evoswitch
Affected IP ranges: Unknown
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

26 aug

Routing problem [Update]

Time: August 26th 2010 16:54:00(CEST)
Expected Downtime: Unknown
Location: TeleCity
Affected IP ranges: Unknown
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in our network in Telecity 2.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[Update 17:00]
We experienced a temporary routing loop which caused problems from our routers at Telecity to some locations. We have fixed the problem and all destinations are reachable again.

26 aug

Routing problem [Update]

Time: August 26th 2010 16:54:00(CEST)
Expected Downtime: Unknown
Location: TeleCity
Affected IP ranges: Unknown
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in our network in Telecity 2.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[Update 17:00]
We experienced a temporary routing loop which caused problems from our routers at Telecity to some locations. We have fixed the problem and all destinations are reachable again.

26 aug

Routing problem [Update]

Time: August 26th 2010 16:54:00(CEST)
Expected Downtime: Unknown
Location: TeleCity
Affected IP ranges: Unknown
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in our network in Telecity 2.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[Update 17:00]
We experienced a temporary routing loop which caused problems from our routers at Telecity to some locations. We have fixed the problem and all destinations are reachable again.

26 aug

Network issues towards Germany and Belgium [Update]

Time: August 25th 2010 20:00:00(CEST)
Expected Downtime: Unknown
Location: TeleCity / Redbus / InterXion / Schuberg Philis / Evoswitch
Affected IP ranges: Unknown
Reason: Fibercut in Germany
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.
One of our fiber providers experiences a fiber outage in Germany.

This affects one of our circuits to DECIX and one circuit to BNIX

You can experience some delay and packet loss when we reroute traffic.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 21:38]
Our fiber provider dispatched engineers to test the fibres and locate the point of the break and they are expected to be onsite to test the fibres at 22:00UTC
We do not expect that the fiber is repaired before tomorrow morning.

[UPDATE 06:15]
Our fiber provider has located the fiber cut between Cologne and Dusseldorf.
The cable system has been damaged in multiple places, the root cause of which is mechanical diggers. Splicing has not been started yet.

[UPDATE 08:30]
Digging works are now complete and splicing work has started.
We expect to be able to provide an estimated time of repair within 60 minutes.

[UPDATE 09:40]
Our fiber provider supplied us with an estimated time to repair. The circuits should be up and running again at 10:30AM.