16 jul

Leaseweb NOC improvement announcement

Startdate: July 16th 2012 07:00:00 (CEST)
Enddate: July 16th 2012 09:00:00 (CEST)

NOC improvement

Dear customer,

Next Monday, 16th of July, we will change the way announcements are posted on our NOC status page.

There used to be only one channel announcing all incidents and maintenance. In some cases this resulted in sending out statuses about products you might not be interested in.

Starting Monday, announcements will be posted in the corresponding categories. You will only receive updates on categories to which you have subscribed. Below you will find an explanation on how to change your subscription and an overview of the different categories.

How to change my subscribed categories?
Currently you are subscribed to all categories. There are two ways to change your subscription:

1. The first option is to change your settings in the Self Service Center. Go to My Account, select Notification settings and choose Network (NOC) from the menu on the left. Behind your listed email address you will find the ‘set categories’ option.
2. The second method is to use the link provided in the NOC announcement e-mails. Each mail will contain a link to change your subscribed categories.

The NOC categorizations explained

• Network: Provides the status for the whole LeaseWeb network infrastructure. All services depend on this network, it is therefore recommended to select this category at all times.
• Datacenter: Gives status updates regarding the datacenter related topics, such as power, cooling and maintenance announcements.
• Mail: Displays all announcements about the LeaseWeb shared mail platform. This is a part of webhosting, batched SMTP and SMTP relay services.
• DNS: Announcements related to our authoritative and resolving name server infrastructure are published here. This infrastructure ensures that your domain zone bought at LeaseWeb is accessible and resolves DNS requests from servers bought at LeaseWeb.
• Express Cloud: Shows the status of the Express Cloud infrastructure and SAN storage.
• Private/Premium cloud: Provides the status of the Private and Premium Cloud infrastructure and SAN storage.
• Webhosting: Announces the status of our shared webhosting platforms for the webhosting product.

If you have any questions regarding this change, please send an email to [email protected].

Kind regards,

LeaseWeb Support

16 jul

[20120717] EvoSwitch high volume router maintenance [POSTPONED] [Update]

Startdate: July 17th 2012 05:00:00 (CEST)
Enddate: July 17th 2012 08:00:00 (CEST)

Dear Customer,

As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.

During this maintenance we will upgrade the software on one of our high volume core routers in EvoSwitch. We've experienced issues before when doing this upgrade, therefore we have planned a maintenance window of 3 hours. During this maintenance we have our vendor standby to assist us.

Affected IP ranges:
85.17.24.0/21
82.192.94.0/23
83.149.124.0/22
62.212.85.0/24
82.192.84.0/23
83.149.70.0/23
95.211.136.0/21
212.7.212.0/23
62.212.82.0/24
62.212.88.0/23
95.211.144.0/20
88.208.57.0/24
46.243.0.0/22
88.208.50.0/23
216.59.55.0/24
88.208.44.0/23
88.208.43.0/24
88.208.52.0/22
88.208.58.0/23
216.59.60.0/24
46.243.4.0/23

2001:1af8:4010::/44

Kind regards,

LeaseWeb – NOC

[20120716 4:00 PM] Due to the issues we have experienced today on the high volume core platform, we are forced to postpone this maintenance. An emergency maintenance will be planned within the same timeframe to upgrade the software version of the router which caused the issues we had today.

16 jul

[20120703]EvoSwitch high volume router maintenance [Update]

Startdate: July 3rd 2012 07:00:00 (CEST)
Enddate: July 3rd 2012 08:00:00 (CEST)

Dear Customer,

As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.

During this maintenance we will change internal routing protocols.

No downtime is expected, however you may notice short packet loss up to 30 minutes within the maintenance window.

Affected IP ranges:
85.17.24.0/21
82.192.94.0/23
83.149.124.0/22
62.212.85.0/24
82.192.84.0/23
83.149.70.0/23
95.211.136.0/21
212.7.212.0/23
62.212.82.0/24
62.212.88.0/23
95.211.144.0/20
88.208.57.0/24
46.243.0.0/22
88.208.50.0/23
216.59.55.0/24
88.208.44.0/23
88.208.43.0/24
88.208.52.0/22
88.208.58.0/23
216.59.60.0/24
46.243.4.0/23

2001:1af8:4010::/44

Kind regards,

LeaseWeb – NOC

[UPDATE 07:01] The maintenance has been performed, all went fine.

16 jul

[Leaseweb] | Wuppertal Network Maintenance [Update]

Startdate: July 4th 2012 05:30:00 (CEST)
Enddate: July 4th 2012 07:30:00 (CEST)

Dear Customer,

****Update-04-07-2012 07:40 CEST:

We have successfully implemented the planned changes. There were roughly 40 minutes of outage. We apologize for the inconvenience caused.

As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.

During the maintenance, we're going to modify the routing setup partially on our edge routers.

We expect an outage during the maintenance, however we will do our utmost to make this downtime as short as possible.

Affected IP subnets:
– 37.1.216.0/21
– 37.58.48.0/20
– 46.166.151.0/24
– 46.166.152.0/21
– 217.20.126.0/23
– 217.20.120.0/22
– 217.20.112.0/21
– 176.65.160.0/21
– 178.162.128.0/17
– 188.72.192.0/18
– 91.191.160.0/21
– 89.149.192.0/18
– 84.16.224.0/19
– 78.159.96.0/19
– 95.168.160.0/19
– 212.95.32.0/19
– 77.223.144.0/23
– 217.73.56.0/21
– 46.166.152.0/21

We do apologize for any inconvenience caused.

Kind regards,

LeaseWeb – NOC

16 jul

EvoSwitch Router hardware upgrade [Update]

Startdate: July 5th 2012 06:00:00 (CEST)
Enddate: July 5th 2012 07:00:00 (CEST)

Dear Customer,

As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.

During this maintenance we will upgrade hardware on one of our standard network distribution routers in EvoSwitch.

Hence this is a redundant platform, the swap of the hardware, will not be done simultaneously on same routers and you should actually see no impact of the maintenance. Since the active router needs to change to the secondary router, there is a loss of connectivity expected for a couple of seconds, but should recover within seconds/minutes.

Affected IP ranges:
85.17.128.0/17
82.192.72.0/21
82.192.92.0/23
95.211.62.0/23
62.212.64.0/20
95.211.32.0/21
62.212.90.0/23
82.192.67.0/24
95.211.48.0/21
95.211.56.0/22
85.17.68.0/22
95.211.64.0/21
82.192.68.0/22
85.17.87.0/24
85.17.88.0/23
85.17.92.0/22
82.192.86.0/24
94.75.226.0/24
91.215.156.0/24
31.186.168.0/23
212.7.204.0/23
95.211.40.0/21
62.212.76.0/22
95.211.164.0/22
62.212.81.0/24

Kind Regards,

Leaseweb

[ UPDATE }

Dear Customer,

The maintenance had been completed,
although you shouldn't have experienced any downtime, please contact our support team if you have any issues.

Kind Regards,

Leaseweb

16 jul

Storage maintenance on Premium and Private/Enterprise Cloud [Update]

Startdate: July 6th 2012 19:00:00 (CEST)
Enddate: July 8th 2012 15:00:00 (CEST)

Dear customer,

LeaseWeb experienced 2 severe outages (dd. 07-01-2012 and 18-03-2012) on the shared storage platform. After investigation by the storage supplier (IBM), it was concluded that these outages were caused by a bug in the operating system. To prevent this outage from happening again, IBM advises to upgrade the system to the latest version.

This maintenance window will be used to upgrade the system to the latest firmware levels.

Affected platforms:
– Premium Cloud
– Private/Enterprise Cloud
– Shared storage customers
Affected Enterprise customers will be informed separately.

No major impact or outage is expected during this maintenance window.

We do apologize for any inconvenience caused.

Kind regards,

LeaseWeb – NOC

[UPDATE 7-07-2012 12:12]

First part of an upgrade is successfully completed. There was no impact for customers. Platform is ready to perform system version upgrade on 8-07-2012]

Kind regards,
LeaseWeb – NOC

[UPDATE 8-07-2012 8:26]

System version upgrade process is started.
LeaseWeb – NOC

[UPDATE 8-07-2012 8:49]

System version upgrade process is in progress.
LeaseWeb – NOC

[UPDATE 8-07-2012 9:15]

First storage node is updated successfully. There was very short disruption, but all services are back online. Right now second storage node is being upgraded.

LeaseWeb – NOC

[UPDATE 8-07-2012 9:30]

Storage platform is updated successfully, and all customer services are online.
Thank you for understanding.
If you have any issues, please contact [email protected]

LeaseWeb – NOC

16 jul

EvoSwitch Router hardware upgrade [Update]

Startdate: July 3rd 2012 06:00:00 (CEST)
Enddate: July 3rd 2012 07:00:00 (CEST)

Dear Customer,

As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.

During this maintenance we will upgrade hardware on one of our standard network distribution routers in EvoSwitch.

Hence this is a redundant platform, the swap of the hardware, will not be done simultaneously on same routers and you should actually see no impact of the maintenance. The hardware on the primary router will be replaced 2 days later.

Affected IP ranges:
85.17.128.0/17
82.192.72.0/21
82.192.92.0/23
95.211.62.0/23
62.212.64.0/20
95.211.32.0/21
62.212.90.0/23
82.192.67.0/24
95.211.48.0/21
95.211.56.0/22
85.17.68.0/22
95.211.64.0/21
82.192.68.0/22
85.17.87.0/24
85.17.88.0/23
85.17.92.0/22
82.192.86.0/24
94.75.226.0/24
91.215.156.0/24
31.186.168.0/23
212.7.204.0/23
95.211.40.0/21
62.212.76.0/22
95.211.164.0/22
62.212.81.0/24

Kind Regards,

Leaseweb

[ UPDATE }

Dear Customer,

The maintenance had been completed,
although you shouldn't have experienced any downtime, please contact our support team if you have any issues.

Kind Regards,

Leaseweb

16 jul

[20120703] EvoSwitch Router software upgrade [Update]

Startdate: July 3rd 2012 06:00:00 (CEST)
Enddate: July 3rd 2012 08:00:00 (CEST)

Dear Sir/Madam,

Leaseweb is announcing a maintenance in order to upgrade the software on one of our high volume routers located in the Evoswitch datacenter (Haarlem, Netherlands).
As the router will be rebooted, your services will be unreachable for up to 30 min during the maintenance window.

The following ranges are impacted:
– 62.212.84.0/24
– 85.17.84.0/24
– 94.75.216.0/23
– 94.75.220.0/22
– 94.75.229.0/24
– 94.75.234.0/24
– 94.75.240.0/24
– 94.75.253.0/24
– 95.211.92.0/23
– 212.7.192.0/22
– 2001:1AF8:4030::/44
– 2A01:9400:3::/48

The maintenance will take place on 3rd of July between 06:00 CET and 08:00 CET (UTC +2)

Kind regards,

LeaseWeb NOC

[Update]
All services are back online and stable. The maintenance window is now closed.

16 jul

Expansion Cloud Express Storage [Update]

Startdate: July 2nd 2012 10:30:00 (CEST)
Enddate: July 2nd 2012 23:59:00 (CEST)

Dear Customer,

On Monday July 2nd we are going to expand our Cloud Express storage.
During this maintenance we are going to place a second storage unit to be able to house more customer instances.
The maintenance will start at 10.30 CEST and is planned to continue until 17.00 CEST.

No impact or outage is expected during this maintenance window.
We expect:
• no outage or reboot on your VM.
• you will be able to perform all actions with your VM.
• you will still be able to access your VM through the Self Service Center.

We do apologize for any inconvenience caused.

Kind regards,

LeaseWeb – NOC

[UPDATE 17:12]
The planned maintenance is going without incidents.

Because the cabling took more time than estimated, we have decided to prolong the maintenance window until 24.00 CEST.

[UPDATE 20:00]
We are experiencing difficulties during the storage expansion. Your virtual machine might lose connection to the storage platform. We are currently working with our vendor to solve the issue as soon as possible.

[UPDATE 20:34]
The storage platform is back online.
We estimate approximately 50 instances were rebooted.
If you are currently still experiencing issues, please contact our support department at [email protected]

[UPDATE 22:45]

The installation of our storage expansion has not been successful today.
At the moment we keep working with our vendor to gather information, we expect to have results on the problems we encountered today.
Based on these results we will take further actions tomorrow from 12.00 CEST onwards.

[UPDATE 15:49]

We successfully have completed the installation of our storage expansion.
We apologize for any inconvenience caused by this upgrade.

16 jul

High Volume Router Issues [Update]

Time: July 16th 2012 14:31:00
Dear Customers,

We are having some high volume router issues in our Haarlem Data Center on the following ranges:

62.212.72.0/23
62.212.82.0/24
62.212.85.0/24
62.212.88.0/23
82.192.94.0/23
82.192.84.0/23
83.149.70.0/23
85.17.24.0/21
83.149.124.0/22
95.211.136.0/21
95.211.144.0/20

We will update this page when more information is available.

[Update 14:41]
The initial cause of the problem was the routing process on the active routing engine.
The process became unresponsive and got killed by the Juniper operating system.
A newly spawned routing process also encountered issues.
We then decided to do a routing engine switchover.
This initially seemed to resolve the issue, but we are finding there are still routing issues.

[Update 14:49]
We are unable to fix the routing process issues on the second routing engine.
The primary routing engine will now be rebooted, once that is back online we will issue another switchover of active routing engine.

[Update 15:04]
We were unable to restore operations on either of the routing engine.
We have decided to initiate a hard reboot of all hardware in the chassis.

[Update 15:12]
Apparently the hard reboot fixed some issues, however the initial issue is still not solved. We are therefore still continue troubleshooting.

[Update 15:31]
Traffic is rebuilding and seems to be restoring. We are investigating the situation and will update this page as soon as there is more information.

[Update 15:42]
Traffic is almost at the same level as before the outage.
If you are still experiencing connectivity issues please open a ticket with support, and clearly mention that you are still experiencing issues.