19 dec

Defective blade [Update]

Time: December 18th 2009 13:35:00(CEST)
Expected Downtime:Unknown
Location: EvoSwitch
Affected ranges : 94.75.218.0/23
94.75.232.0/24
94.75.250.0/24

Reason: Hardware failure
*******************************

Dear Customer,

At the moment we experience some troubles with one of our modules in one of our routers.

We are currently replacing this module.
Customers in the mentioned ranges may experience network issues. This will be restored ASAP.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 13.50]

The module has been replaced by our engineers.
All services should be restored. We will closely monitor the blade, for any issues the upcoming hour.

If you still experience downtime of your services, please email support so we can bring your service back online.

Regards,
LeaseWeb BV

[UPDATE 14.50]
All services have been restored.

[UPDATE 17.30]
We are still seeing strange behaviour on this router, we are currently busy to stabalize this issue. When we are not able to stabalize this issue we need to do a reload. If we have any news we will update the NOC page.

[update 17.38]
We are unable to stabalize the issue, we are going to do an emergency reload with a new software version.

[update 18.00]
The router is online again after the reload, we will investigate if all traffic will restore correctly.

[update 18.15]
The router is up for 15 minutes, and we see the traffic restoring to previous values. Please open a ticket if you still have a problem with reaching your server.

Regards,
LeaseWeb BV

17 dec

Backup peering connections maintenance

Startdate: December 20th 2009 00:01:00(CEST)
Enddate: December 20th 2009 05:00:00(CEST)

Dear Customer,

Our provider announced a maintenance of their backup infrastructure acting as a hub for some of our links to European internet exchanges. As only the backup circuits may be down within the maintenance window no impact is expected.


The timezone is CET (GMT+1).

Kind regards,

Leaseweb NOC

14 dec

router linecard move

Startdate: December 15th 2009 09:00:00(CEST)
Enddate: December 15th 2009 10:00:00(CEST)

Dear Customer,

During this window we will move two linecards to different slots in our core router at Evoswitch.
There should be no impact on traffic.

Kind regards,

Leaseweb NOC

04 dec

Frankfurt and Stockholm connection maintenance

Startdate: December 13th 2009 00:01:00(CEST)
Enddate: December 13th 2009 06:00:00(CEST)

Dear customer,

Two of our circuits to DE-CIX (German Internet Exchange) and one to Netnod (Swedish Internet Exchange) will be maintained.

No impact is expected as we will reroute our traffic to other links. During traffic switching and tuning there may be small temporary packet loss or latency increase.

The timezone is CET (GMT+1).

Kind regards,

Leaseweb NOC

04 dec

DE-CIX connection maintenance

Startdate: December 4th 2009 00:00:00(CEST)
Enddate: December 4th 2009 03:00:00(CEST)

Dear customer,

Two of our circuits to DE-CIX (German Internet Exchange) will be maintained.

No impact is expected as we will reroute our traffic to other links. During traffic switching and tuning there may be small temporary packet loss or latency increase.

The timezone is CET (GMT+1).

Kind regards,

Leaseweb NOC

04 dec

Core router upgrade [Update]

Startdate: December 7th 2009 10:00:00(CEST)
Enddate: December 7th 2009 11:00:00(CEST)

Dear Customer,

During this maintenance window we will upgrade the core router at Evoswitch.
We will add interface modules.
After the upgrade we will have 16 extra 10GE ports available.

There is no impact expected.
The router supports O.I.R. (online insertion and removal)

Kind regards,

LeaseWeb NOC

[UPDATE]
I’m sorry to report I made a mistake when entering the maintenance date.
The correct date is the eighth of December 2009.

[UPDATE]
We’ve received notification from our distributor.
Delivery of the new modules has been postponed.
As such we have moved this maintenance window forward as well.

[UPDATE]
Delivery has been pushed forward, therefore the maintenance window has been brought back to Monday 7th of December.

04 dec

LINX connection maintenance [Update]

Startdate: November 30th 2009 22:00:00(CEST)
Enddate: December 1st 2009 09:00:00(CEST)

Dear customer,

One of our circuits to LINX (London Internet Exchange) will be maintained. No impact is expected as we will reroute our traffic to other links. During traffic switching and tuning there may be small temporary packet loss or rtt increase.

The timezone is CET (GMT+1).

Kind regards,

Leaseweb NOC


[Update 27.11.2009 12:00]
We have just received emergency update from our carrier: there is a possibility the fiber backbone rerouting may take them longer than previously expected, so in the worst case this maintenance may last even to 9:00 AM CET.
As we have backup capacity on other links we do not expect any serious impact on our network.

03 dec

UPS Maintenance

Time: December 3rd 2009 16:00:00(CEST)
Expected Downtime: 1 Hour
Location: Evo-Switch
Affected IP ranges: HAL1 en partial HAL2
Reason: UPS disruption
*******************************

Dear customer,

The EvoSwitch datacenter scheduled an 16h00 – 17h00 emergency maintenance window after todays UPS disruption. We expect no services to be affected during this maintenance.

We apologize for the short notice and any inconvenience caused

Regards,
LeaseWeb BV

03 dec

EvoSwitch Hall 1/2 problems [Update]

Time: December 3rd 2009 02:05:00(CEST)
Expected Downtime: Unknown
Location: EvoSwitch
Reason: under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 02:20] All services are back online again, we are currently investigating the cause if this issue.

[UPDATE 02:58] It seems some feeds had a short power dip. This is not certain yet. We are still investigating.

[UPDATE 03:12] Some colocation customers are causing ethernet loops, these interfaces are being shut now.

[UPDATE 03:17] to clarify, the loops did not start the problems, it seems that the colocation customers switches are causing loops after the power dip.
All looping interfaces have been shutdown.

[UPDATE 03:39] Four extra engineers are at the datacenter. And some more underway. If your server is still not online please contact support via email.

[UPDATE 07:20] A customer in Hall 2 who was replacing network equipment caused a loop which caused partial unreachability in Hall2.

[UPDATE 07:40] The customer who caused the loop has been taken offline and all services have been restored.

03 dec

EvoSwitch Hall 1/2 problems [Update]

Time: December 3rd 2009 02:05:00(CEST)
Expected Downtime: Unknown
Location: EvoSwitch
Reason: under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 02:20] All services are back online again, we are currently investigating the cause if this issue.

[UPDATE 02:58] It seems some feeds had a short power dip. This is not certain yet. We are still investigating.

[UPDATE 03:12] Some colocation customers are causing ethernet loops, these interfaces are being shut now.

[UPDATE 03:17] to clarify, the loops did not start the problems, it seems that the colocation customers switches are causing loops after the power dip.
All looping interfaces have been shutdown.

[UPDATE 03:39] Four extra engineers are at the datacenter. And some more underway. If your server is still not online please contact support via email.

[UPDATE 07:20] A customer in Hall 2 who was replacing equipment caused a loop.

[UPDATE 07:40] The customer who caused the loop has been taken offline