Startdate: July 6th 2012 19:00:00 (CEST)
Enddate: July 8th 2012 15:00:00 (CEST)

Dear customer,

LeaseWeb experienced 2 severe outages (dd. 07-01-2012 and 18-03-2012) on the shared storage platform. After investigation by the storage supplier (IBM), it was concluded that these outages were caused by a bug in the operating system. To prevent this outage from happening again, IBM advises to upgrade the system to the latest version.

This maintenance window will be used to upgrade the system to the latest firmware levels.

Affected platforms:
– Premium Cloud
– Private/Enterprise Cloud
– Shared storage customers
Affected Enterprise customers will be informed separately.

No major impact or outage is expected during this maintenance window.

We do apologize for any inconvenience caused.

Kind regards,

LeaseWeb – NOC

[UPDATE 7-07-2012 12:12]

First part of an upgrade is successfully completed. There was no impact for customers. Platform is ready to perform system version upgrade on 8-07-2012]

Kind regards,
LeaseWeb – NOC

[UPDATE 8-07-2012 8:26]

System version upgrade process is started.
LeaseWeb – NOC

[UPDATE 8-07-2012 8:49]

System version upgrade process is in progress.
LeaseWeb – NOC

[UPDATE 8-07-2012 9:15]

First storage node is updated successfully. There was very short disruption, but all services are back online. Right now second storage node is being upgraded.

LeaseWeb – NOC

[UPDATE 8-07-2012 9:30]

Storage platform is updated successfully, and all customer services are online.
Thank you for understanding.
If you have any issues, please contact [email protected]

LeaseWeb – NOC

Startdate: July 2nd 2012 10:30:00 (CEST)
Enddate: July 2nd 2012 23:59:00 (CEST)

Dear Customer,

On Monday July 2nd we are going to expand our Cloud Express storage.
During this maintenance we are going to place a second storage unit to be able to house more customer instances.
The maintenance will start at 10.30 CEST and is planned to continue until 17.00 CEST.

No impact or outage is expected during this maintenance window.
We expect:
• no outage or reboot on your VM.
• you will be able to perform all actions with your VM.
• you will still be able to access your VM through the Self Service Center.

We do apologize for any inconvenience caused.

Kind regards,

LeaseWeb – NOC

[UPDATE 17:12]
The planned maintenance is going without incidents.

Because the cabling took more time than estimated, we have decided to prolong the maintenance window until 24.00 CEST.

[UPDATE 20:00]
We are experiencing difficulties during the storage expansion. Your virtual machine might lose connection to the storage platform. We are currently working with our vendor to solve the issue as soon as possible.

[UPDATE 20:34]
The storage platform is back online.
We estimate approximately 50 instances were rebooted.
If you are currently still experiencing issues, please contact our support department at [email protected]

[UPDATE 22:45]

The installation of our storage expansion has not been successful today.
At the moment we keep working with our vendor to gather information, we expect to have results on the problems we encountered today.
Based on these results we will take further actions tomorrow from 12.00 CEST onwards.

[UPDATE 15:49]

We successfully have completed the installation of our storage expansion.
We apologize for any inconvenience caused by this upgrade.

Time: July 11th 2012 22:22:00
Dear Customer,
We are currently experiencing issues with small part of our Express Cloud storage platform.
If your server became unreachable within last 5 minutes, it means that probably you are affected by this.
We are now working on solving this issue.
You will be notified shortly about status of this issue.

Kind regards,
LeaseWeb – NOC

[UPDATE 22:46]
Affected storage server is back online, however there is still problem with increased disk latency, and this issues is now under investigation, so your VPS (if it is affected by this problem) may experience slow performance right now.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-11 22:57]

We are now replacing network cable and switchport in affected storage system.
We will update you shortly if problem is solved by this change.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-11 23:27]

Unfortunately replacing network cable and switchport didn't solve this problem.
We are going to replace network card now to allow further investigation.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-12 0:09]

Replacing network card didn't solve this problem, so now we are performing detailed system diagnosis to verify if this is hardware related problem.
All affected VPS'es are unreachable right now.
Please be patient while we are working on this issue.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-12 0:47]

Hardware components will be relpaced very soon. Right now part of affected VPS'es is online, but with very low performance. After hardware is replaced, we expect performance to be back to normal level.
Thank you for your patience while we work on this issue.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-12 0:58]

Performance of this storage server is better now and affected VPS'es should be back online. It's not yet optimal, but allows to keep customer VPS'es online and reachable.
At this moment we are still fixing some VPS'es that require manual intervention, so if you experience any issues with your server, please contact [email protected]

Thank you for your patience.

Kind regards,
LeaseWeb – NOC

Time: July 11th 2012 22:22:00
Dear Customer,
We are currently experiencing issues with small part of our Express Cloud storage platform.
If your server became unreachable within last 5 minutes, it means that probably you are affected by this.
We are now working on solving this issue.
You will be notified shortly about status of this issue.

Kind regards,
LeaseWeb – NOC

[UPDATE 22:46]
Affected storage server is back online, however there is still problem with increased disk latency, and this issues is now under investigation, so your VPS (if it is affected by this problem) may experience slow performance right now.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-11 22:57]

We are now replacing network cable and switchport in affected storage system.
We will update you shortly if problem is solved by this change.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-11 23:27]

Unfortunately replacing network cable and switchport didn't solve this problem.
We are going to replace network card now to allow further investigation.

Kind regards,
LeaseWeb – NOC

Time: July 11th 2012 22:22:00
Dear Customer,
We are currently experiencing issues with small part of our Express Cloud storage platform.
If your server became unreachable within last 5 minutes, it means that probably you are affected by this.
We are now working on solving this issue.
You will be notified shortly about status of this issue.

Kind regards,
LeaseWeb – NOC

[UPDATE 22:46]
Affected storage server is back online, however there is still problem with increased disk latency, and this issues is now under investigation, so your VPS (if it is affected by this problem) may experience slow performance right now.

Kind regards,
LeaseWeb – NOC

[UPDATE 2012-07-11 22:57]

We are now replacing network cable and switchport in affected storage system.
We will update you shortly if problem is solved by this change.

Kind regards,
LeaseWeb – NOC

Time: July 8th 2012 10:07:00
Dear customer,

Currently LeaseWeb is working on fixing issues on the Premium Cloud platform, related to storage network. The impact of the issue is limited to part of the Premium Cloud instances

Kind Regards,

LeaseWeb – NOC

[UPDATE 8-07-2012 10:32]

Problems have been solved. Customers who had problems rebooting their VPS should be able to reboot their VPS again. If you still encounter problems please send an email to [email protected] so we can help you solve them.

Kind Regards,

LeaseWeb – NOC

Time: July 1st 2012 12:44:00
Dear customer,

Currently LeaseWeb is working on fixing issues on the Express Cloud platform, caused by the 'leap second bug' that is impacting various platforms world-wide.
The impact of the issue is limited to part of the Express Cloud instances. The following services might not function correctly:
– Stopping, starting and rebooting of instances via the SSC
– Editing firewall rules and switching the firewall on/off
Currently we are working on fixing these issues. We might need to restart a small amount of instances during the maintenance.
We do apologize for any inconvenience caused.
Kind regards,

LeaseWeb – NOC

UPDATE 20120701 13:30

We have restarted a number of management processes. We are now monitoring our systems, and waiting for a number of queued tasks to finish, and for the system to stabilize further. As the accumulated number of queued tasks is quite high, the impacted functionality is not fully available yet. We have not needed to restart any customer instances.

UPDATE 20120701 15:35

The platform is still stabilizing. Due to the large number of concurrent jobs running in the system, we have had to restart a small amount of instances (currently less than 10) to help bring the system back in sync. The platform is now functioning for part of the customer base, but operations on instances via the SSC can take a while to complete. We continue to work to get all services back online as soon as possible.

Kind regards,

LeaseWeb – NOC

UPDATE 20120701 16:30

The platform has stabilized, and all SSC features are working again. Due to increased activity on the platform, it might take a little longer for some actions to complete. Total impact on customer instances has been limited to roughly 10 restarted instances. In case of any issues, please contact us via the support system. We again apologize for the inconvenience.

Kind regards,

LeaseWeb – NOC

Time: July 1st 2012 12:44:00
Dear customer,

Currently LeaseWeb is working on fixing issues on the Express Cloud platform, caused by the 'leap second bug' that is impacting various platforms world-wide.
The impact of the issue is limited to part of the Express Cloud instances. The following services might not function correctly:
– Stopping, starting and rebooting of instances via the SSC
– Editing firewall rules and switching the firewall on/off
Currently we are working on fixing these issues. We might need to restart a small amount of instances during the maintenance.
We do apologize for any inconvenience caused.
Kind regards,

LeaseWeb – NOC

UPDATE 20120701 13:30

We have restarted a number of management processes. We are now monitoring our systems, and waiting for a number of queued tasks to finish, and for the system to stabilize further. As the accumulated number of queued tasks is quite high, the impacted functionality is not fully available yet. We have not needed to restart any customer instances.

UPDATE 20120701 15:35

The platform is still stabilizing. Due to the large number of concurrent jobs running in the system, we have had to restart a small amount of instances (currently less than 10) to help bring the system back in sync. The platform is now functioning for part of the customer base, but operations on instances via the SSC can take a while to complete. We continue to work to get all services back online as soon as possible.

Kind regards,

LeaseWeb – NOC

Time: July 1st 2012 12:44:00
Dear customer,

Currently LeaseWeb is working on fixing issues on the Express Cloud platform, caused by the 'leap second bug' that is impacting various platforms world-wide.
The impact of the issue is limited to part of the Express Cloud instances. The following services might not function correctly:
– Stopping, starting and rebooting of instances via the SSC
– Editing firewall rules and switching the firewall on/off
Currently we are working on fixing these issues. We might need to restart a small amount of instances during the maintenance.
We do apologize for any inconvenience caused.
Kind regards,

LeaseWeb – NOC

UPDATE 20120701 13:30

We have restarted a number of management processes. We are now monitoring our systems, and waiting for a number of queued tasks to finish, and for the system to stabilize further. As the accumulated number of queued tasks is quite high, the impacted functionality is not fully available yet. We have not needed to restart any customer instances.

Time: July 1st 2012 12:44:00
Dear customer,

Currently LeaseWeb is working on fixing issues on the Express Cloud platform, caused by the 'leap second bug' that is impacting various platforms world-wide.
The impact of the issue is limited to part of the Express Cloud instances. The following services might not function correctly:
– Stopping, starting and rebooting of instances via the SSC
– Editing firewall rules and switching the firewall on/off
Currently we are working on fixing these issues. We might need to restart a small amount of instances during the maintenance.
We do apologize for any inconvenience caused.
Kind regards,

LeaseWeb – NOC