Startdate: February 28th 2012 06:00:00(CEST)
Enddate: February 28th 2012 07:00:00(CEST)

Dear Customer,



As part of our commitment to continually improve the level of service you receive from LeaseWeb we are undertaking this maintenance.



During this maintenance window we will replace routing engines of one of our high volume routers.



You may notice short packet loss or downtime up to 15 minutes within the maintenance window.



Affected IP ranges:



62.212.72.0/23

62.212.82.0/24

62.212.85.0/24

62.212.88.0/23

82.192.94.0/23

83.149.70.0/23

83.149.124.0/22

85.17.24.0/21

95.211.136.0/21

95.211.144.0/20

2001:1af8:4010::/44



Kind regards,



LeaseWeb – NOC





[UPDATE 7:15]

Faulty linecard blocked router fabric and caused some destination to be unreachable. The linecard was swapped before the maintenance window had ended. Please open a support ticket should you see any further problems.

Startdate: February 27th 2012 00:00:00(CEST)
Enddate: March 20th 2012 00:00:00(CEST)

Introduction:



The Schuberg Philis datacenter is being prepared for a major change in its cooling system. This change will ensure higher cooling efficiency within the datacenter, as well as more future capacity available within the datacenter.



At the time of the maintenance, parts of the cooling system will be reconnected to new cooling units. During the actual maintenance, limited cooling capacity will be available for the datacenter room which is being upgraded. Since the Schuberg Philis datacenter is fully redundant in terms of cooling systems and cooling capacity, the datacenter will make use of this redundancy to perform the maintenance.



Cooling redundancy and capacity tests for each dataroom have already been executed successfully. Schuberg Philis does not expect any cooling issues during the maintenance, but still they have taken additional measures in case of unexpected cooling issues during the maintenance. (E.g. failure of a datacenter room pump, etc).



The maintenance will take place in February and March of 2012.



Schedule:



27th of february ( four airco’s )

2nd of march ( four airco’s )

20th of march ( four airco’s )

14th march ( four airco’s )



——UPDATE——



All maintenance windows are completed successfully without unexpected cooling issues.

Time: March 28th 2012 16:12:00(CEST)
Update: 17:50



Cause was identified and we are taking steps in addressing the issue.



Update: 17:15



Problem was isolated and we are taking steps in restoring full network stability.

A small number of servers from the initially affected section are currently unreachable.

Next Update at 17:45



Update: 17:00



Investigation is still ongoing, next update in 30min.

Sorry for the inconvenience caused.



Update since 22:00



The issue have been resolved and everything should be up-and-running. Also steps have been taking for possible events in the future too address this sooner. If you still have issues, please contact us at [email protected]



Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: see below

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on a small part of our network in Frankfurt.

A small portion of our customers can experience service degradation or service interruption.

We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV

Time: March 27th 2012 20:58:00(CEST)
Final Update: 28-03-2012 12:00

The root cause of the network issue was due to very large lldp multicast loop-flood.

This type of network issue was very difficult to detect and took a long time to pin-point. We will in the near future plan a maintenance in to replace some hardware that will restrict this type of outage.



Our apologies for any inconvenience caused.



Update: 23:10



Currently 95% of the connected customers are back online, we are still investigating a small network segment.

We will continue to make updates as new information is available.



Update: 22:20



We have reduced the network issue down and 80% of all affected server are back online.

We are now trouble shooting the issue on the last affect segment.

We will continue to make updates as new information is available.







Update: 22:00



Investigation is still ongoing, the cause the network outage is still not known.

We will continue to make updates as new information is available.



Update: 21:25



Network engineer is onsite, still investigating the issue.

We will continue to make updates as new information is available.







Sorry for the inconvenience caused.





Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: see below

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on part of our network in Frankfurt



We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV



*********************************************************************************

German Translation:



Sehr geehrte Kunden,



momentan haben wir Probleme in Teilen unseres Netzwerkes in Frankfurt.



Wir werden das Beste machen, um das Problem schnellstmöglichst zu beheben.

Sobald wir neue Informationen haben, werden wir sie hier für Sie bereitstellen.



Folgende IP – Adressbereiche sind von dem Ausfall betroffen:



1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Bitte gedulden Sie sie noch etwas während wir an dem Problem arbeiten.



Mit freundlichen Grüßen



Leaseweb Germany GmbH

Time: March 27th 2012 20:58:00(CEST)
Update: 23:10



Currently 95% of the connected customers are back online, we are still investigating a small network segment.

We will continue to make updates as new information is available.



Update: 22:20



We have reduced the network issue down and 80% of all affected server are back online.

We are now trouble shooting the issue on the last affect segment.

We will continue to make updates as new information is available.







Update: 22:00



Investigation is still ongoing, the cause the network outage is still not known.

We will continue to make updates as new information is available.



Update: 21:25



Network engineer is onsite, still investigating the issue.

We will continue to make updates as new information is available.



Update: 23:10







Sorry for the inconvenience caused.





Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: see below

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on part of our network in Frankfurt



We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV



*********************************************************************************

German Translation:



Sehr geehrte Kunden,



momentan haben wir Probleme in Teilen unseres Netzwerkes in Frankfurt.



Wir werden das Beste machen, um das Problem schnellstmöglichst zu beheben.

Sobald wir neue Informationen haben, werden wir sie hier für Sie bereitstellen.



Folgende IP – Adressbereiche sind von dem Ausfall betroffen:



1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Bitte gedulden Sie sie noch etwas während wir an dem Problem arbeiten.



Mit freundlichen Grüßen



Leaseweb Germany GmbH

Time: March 27th 2012 20:58:00(CEST)
Update: 22:20



We have reduced the network issue down and 80% of all affected server are back online.

We are now trouble shooting the issue on the last affect segment.

We will continue to make updates as new information is available.







Update: 22:00



Investigation is still ongoing, the cause the network outage is still not known.

We will continue to make updates as new information is available.



Update: 21:25



Network engineer is onsite, still investigating the issue.

We will continue to make updates as new information is available.



Sorry for the inconvenience caused.





Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: see below

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on part of our network in Frankfurt



We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV



*********************************************************************************

German Translation:



Sehr geehrte Kunden,



momentan haben wir Probleme in Teilen unseres Netzwerkes in Frankfurt.



Wir werden das Beste machen, um das Problem schnellstmöglichst zu beheben.

Sobald wir neue Informationen haben, werden wir sie hier für Sie bereitstellen.



Folgende IP – Adressbereiche sind von dem Ausfall betroffen:



1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Bitte gedulden Sie sie noch etwas während wir an dem Problem arbeiten.



Mit freundlichen Grüßen



Leaseweb Germany GmbH

Time: March 27th 2012 17:00:00(CEST)
Dear Customer,



Regrettably we have to announce that one of our datagraph servers corrupted graph data, we are currently still restoring backups.



As a result it looks like we lost about 2,5 weeks of data in the worst case scenario.



Affected customers:

-Colocation customers in the Netherlands

-Some high volume customers in the Netherlands



Our apologies for any inconvenience caused.



Please note that this issue is not service affecting.



Kind regards,



LeaseWeb – NOC

Time: March 27th 2012 20:58:00(CEST)
Update: 21:25



Network engineer is onsite, still investigating the issue.

We will continue to make updates as new information is available.



Sorry for the inconvenience caused.





Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: see below

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on part of our network in Frankfurt



We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV



*********************************************************************************

German Translation:



Sehr geehrte Kunden,



momentan haben wir Probleme in Teilen unseres Netzwerkes in Frankfurt.



Wir werden das Beste machen, um das Problem schnellstmöglichst zu beheben.

Sobald wir neue Informationen haben, werden wir sie hier für Sie bereitstellen.



Folgende IP – Adressbereiche sind von dem Ausfall betroffen:



1 212.95.32.0/19

2 84.16.224.0/19

4 95.168.160.0/19

5 78.159.96.0/19

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Bitte gedulden Sie sie noch etwas während wir an dem Problem arbeiten.



Mit freundlichen Grüßen



Leaseweb Germany GmbH

Time: March 27th 2012 20:58:00(CEST)
Expected Downtime: Unknown

Location: Frankfurt Databurg

Affected IP ranges: 62.212.64.xxx – 62.212.64.xxx / Unknown

Under investigation / Unknown

*******************************

Dear customer,



At the moment we experience some troubles on part of our network in Frankfurt



We are doing our utmost to solve this issue a.s.a.p.

As soon as we have more information available, we will publish it here.



Affected ranges

1 212.95.32.0/19

2 84.16.224.0/19

3 217.20.112.0/20 le 23

4 95.168.160.0/19

5 78.159.96.0/19

6 89.149.192.0/18 le 24

7 188.72.192.0/18

8 91.191.160.0/21

10 178.162.128.0/17

11 77.223.144.0/23

12 46.166.152.0/21

14 217.73.56.0/21

15 176.65.160.0/21

16 46.166.151.0/24

17 37.58.48.0/20

18 37.1.216.0/21



Please be patient while we are working on this problem.



Regards,

LeaseWeb BV