We will perform a software upgrade on one (1) standard (primary) HSRP router in the LOFT (US) datacenter. Since we use HSRP to provide redundancy on the standard network there is no impact, however during the swap from standby to active you can notice some packet-loss. If you have a private rack please make sure that redundancy is working correctly.

The affected ranges are below:
Standard network
– 108.59.0.0/24
– 108.59.1.0/24
– 108.59.2.0/24
– 108.59.5.0/24
– 108.59.6.0/23
– 108.59.15.0/24
– 108.59.12.0/24

The maintenance will take place on the 28th of October between 12:00 -06:00 EDT Manassas- and 12:30 CEST -06:30 EDT Manassas- Time.

We will perform a software upgrade on four (4) standard (primary) HSRP routers in the Evoswitch datacenter. Since we use HSRP to provide redundancy on the standard network there is no impact, however during the swap from standby to active you can notice some packet-loss. If you have a private rack please make sure that redundancy is working correctly.

The affected ranges are below:
Standard network, number one
95.211.48.0/21
95.211.56.0/23
62.212.88.0/23
85.17.68.0/22
85.17.135.0/24
85.17.181.0/24

Standard network, number two
95.211.40.0/21
95.211.58.0/23
95.211.62.0/23
95.211.64.0/21
62.212.67.0/24
62.212.74.0/24
62.212.91.0/24
82.192.86.0/24
82.192.71.0/24
85.17.134.0/24
85.17.151.0/24
85.17.199.0/24
85.17.202.0/23
85.17.204.0/23
85.17.206.0/24
85.17.219.0/24
85.17.226.0/24
85.17.229.0/24
85.17.231.0/24
85.17.232.0/24
85.17.235.0/24
85.17.236.0/23
85.17.238.0/24
85.17.242.0/23
31.186.168.0/23

Standard network, number three
– 94.75.194.0/24
– 94.75.195.0/24
– 94.75.196.0/24
– 94.75.197.0/24
– 94.75.200.0/21
– 94.75.208.0/21
– 94.75.224.0/23
– 94.75.228.0/24
– 94.75.233.0/24
– 94.75.240.0/20
– 95.211.0.0/19
– 212.7.192.128/25
– 212.7.193.128/25
– 212.7.208.0/24
– 94.124.56.0/21
– 62.212.75.0/24
– 87.236.96.0/21
– 90.156.224.0/20
– 82.192.87.0/24
– 82.192.88.0/24
– 46.182.176.0/22
– 193.107.18.0/24
– 46.166.148.0/23
– 46.166.150.0/24

Standard network, number four
– 95.211.72.0/21
– 95.211.108.0/22
– 95.211.112.0/20
– 212.7.200.0/22
– 95.211.134.0/23
– 95.211.106.0/23
– 82.192.74.0/23
– 82.192.92.0/23
– 95.211.160.0/23
– 82.192.66.0/24
– 31.186.170.0/24
– 83.149.102.0/23
– 188.95.136.0/22
– 94.177.98.0/23
– 92.114.86.0/23
– 82.192.90.0/24
– 212.7.214.0/24
– 46.226.192.0/22

The maintenance will take place on the 25th of October between 06:00 and 07:00 CEST.

Time: October 13th 2011 13:44:00(CEST)
Expected Downtime: Unknown

Location: Wupertal Germany

Reason: Hardware failure

*******************************

Dear Customer,

At the moment we experience some troubles in the German part of our network, datacenter Wupertal. The cause of this is that a link between the XMR and the core went down.

We are currently in contact with the provider to have an engineer onsite to resolve the issue as soon as possible.
Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

UPDATE @ 17:40
We located the faulty segment on the circuit. The engineer is now on the way to the site to measure and repair the segment.

UPDATE @ 18:50
The engineer is stuck in the traffic so it is taking a bit longer then expected. We can expect him to arrive within 1 hour

UPDATE @ 19:30
The engineer has arrived. Now locating the fault and repair

UPDATE @ 20:30
Broken Fiber Strand found, checking fastest possible solution for fix

UPDATE @ 20:50
Splicing Team is on there way to remove broken part and splice in a new part

UPDATE @ 21:55
Fiber Strand is not reparable, new cable is beeing pulled

UPDATE @ 23:40
around 200 Meters cable pulled, pigtail splicing begins shortly

UPDATE @ 02:15
Everything is back online

=====

German Translation:

Sehr geehrte Kunden,

Im Moment haben wir ein Netzwerkproblem in einem Teil unseres Deutschen Netzwerks im Datacenter in Wuppertal. Der Grund ist, dass eine Leitung zwischen dem Core und der XMR down ist.

Wir sind in Kontakt mit dem Provider und haben einen Techniker vor Ort um das Problem so schnell wie moeglich zu beheben.
Wir bedanken uns fuer Ihr Verstaendnis und Ihre Geduld.

Mit freundlichen Gruessen,
LeaseWeb Germany GmbH

UPDATE @ 17:40
Das fehlerhafte Kabelsegment wurde lokalisiert, ein Techniker des Leitungsbetreibers ist auf dem Weg zum Rechenzentrum um den Fehler zu beheben

UPDATE @ 18:50
Der Techniker verspaetet sich aufgrund des Verkehrsaufkommens, wir erwarten ihm laufe der naechsten Stunde

UPDATE @ 19:30
Der Techniker ist angekommen. Fehlerbehebung beginnt

UPDATE @ 20:30
Das defekte Kabel und die Schadstelle sind lokalisiert, zur Zeit wird nach der schnellsten Loesung gesucht um das Problem zu behoben

UPDATE @ 20:50
Ein Splicing Team ist unterwegs um das betroffene Teilstueck zu entfernen und ein neues einzusetzen

UPDATE @ 21:55
Das defekte Kabel ist irreperabel Beschaedigt, ein neues Kabel wird zur Zeit eingezogen

UPDATE @ 23:40
Knapp 200Meter Kabel wurden neu verlegt, das splicen der Verbinder startet in kuerze

UPDATE @ 02:15
Alle Leitung sind repariert, Arbeiten abgeschlossen

We will perform a software upgrade on two of the standby routers in the Evoswitch datacenter. Since we use HSRP to provide redundancy there should be no impact because of this. If you have a private rack please make sure that redundancy is working correctly.

The following ranges are impacted for the first router:
95.211.48.0/21
95.211.56.0/23
62.212.88.0/23
85.17.68.0/22
85.17.135.0/24
85.17.181.0/24
178.18.21.0/23
178.18.22.0/24
193.27.73.252
193.27.72.0/23

The following for the second router:
91.215.156.0/24
31.186.168.0/23
62.212.67.0/24
62.212.74.0/24
62.212.91.0/24
82.192.86.0/24
82.192.71.0/24
85.17.134.0/24
85.17.151.0/24
85.17.199.0/24
85.17.202.0/23
85.17.204.0/23
85.17.206.0/24
85.17.219.0/24
85.17.226.0/24
85.17.229.0/24
85.17.231.0/24
85.17.232.0/24
85.17.235.0/24
85.17.236.0/23
85.17.238.0/24
85.17.242.0/23
93.190.220.0/24
94.75.226.0/24
94.75.239.0/24
95.211.40.0/21
95.211.58.0/23
95.211.62.0/23
95.211.64.0/21

The maintenance will take place on 18 october between 06:00 CEST and 07:00 CEST (UTC +2)

Kind regards,
LeaseWeb NOC

[UPDATE @ 06:15]
The upgrade of both routers is done and was a success. For both routers the interface status and standby status for HSRP have been verified. These were the same as before and OK.

Dear customer,

Last night during a failover of the firewall we had connectivity issues to customers behind the firewall. To resolve this issue we had to revert back to previous active firewall.

We have identified the probable cause of this interruption and have fixed the issue. To be sure we have scheduled this maintenance to force a failover and check if all traffic to customers behind the firewall will continue as normal.

We expect no impact during the failover. In case of problems we will revert to the previous active firewall.

The ranges on the firewall are:
85.17.243.192/27
94.75.198.160/27
95.211.65.0/25
95.211.65.128/26
95.211.67.192/27
95.211.68.0/27
95.211.69.0/27
95.211.71.128/27

Kind regards,
Leaseweb NOC

[Update 00:05]
We have done the failover and are checking if all servers are online. If you have any issues please contact us at [email protected]. We will revert back to the normal situation in 5 minutes.If we see any issues we will restore immediately.

[Update 00:10]
We have restored to the normal situation. The failover worked as expected.

Dear customer,

Please note that due to maintenance, it???s possible www.leaseweb.com and secure.leaseweb.com will not be reachable within the given timeframe.
The maintenance will take place on Tuesday October 11 between 00:00 and 05:00 (UTC/GMT +2 hours).

Kind Regards,
LeaseWeb Support

Startdate: October 6th 2011 05:00:00(CEST)
Enddate: October 6th 2011 07:00:00(CEST)

Dear Customer,



On the 5th of October, during a planned maintenance, we installed an upgraded version of IOS-XR software on our CRS core router at EvoSwitch.

The installed version was intended to be an intermediate version.

As IOS-XR upgrades are lengthy we split up the upgrades into multiple maintenance windows.



The version that is currently running is showing random issues with forwarding traffic. The impact, as we currently see it, is minimal. Only a small amount of prefixes is affected. Once we re-learn the prefixes, the problem is solved.



However it is not a situation we want to maintain until the next planned upgrade.

As such we will perform the next phase of software upgrade during this emergency maintenance window.



During the maintenance you might notice some degredation in performance while alternative routes are learned.



Our apologies for any inconvenience this might cause.





Kind regards,



LeaseWeb NOC.

Time: October 5th 2011 13:45:00(CEST)
Dear Customer,



We received some reports about unreachable IP’s after the core router maintenance we did this morning. Currently we are still investigating.



If you experience connectivity troubles, could you please send us bi-directional traceroute results so we can investigate further? Bi-directional means from the external location to the LeaseWeb IP, and from the LeaseWeb IP back to the external location.



Thanks in advance.



Regards,

LeaseWeb BV



[UPDATE 14:55] The issue has been resolved, if you still see traffic being blackholed please contact us.



Sehr verehrte Kunden,



nach der Core Router Maintenance von heute Morgen haben wir vereinzelt Meldungen über nicht erreichbare IP Netze. Dies wird zur Zeit von uns geprüft.



Sollten sie Probleme bei der Erreichbarkeit feststellen möchten wir sie bitten uns sog. Traceroutes von/zu ihrem Server bei uns zu senden. Basierend auf diesen Daten kann eine Analyse erfolgen.



Mit freundlichen Grüssen



LeaseWeb



[UPDATE 14:55] Das Problem wurde behoben. Sollten sie immer noch Probleme mit der Erreichbarkeit haben bitte wir Sie dies uns mitzuteilen

Time: October 5th 2011 13:45:00(CEST)
Dear Customer,



We received some reports about unreachable IP’s after the core router maintenance we did this morning. Currently we are still investigating.



If you experience connectivity troubles, could you please send us bi-directional traceroute results so we can investigate further? Bi-directional means from the external location to the LeaseWeb IP, and from the LeaseWeb IP back to the external location.



Thanks in advance.



Regards,

LeaseWeb BV



[UPDATE 14:55] The issue has been resolved, if you still see traffic being blackholed please contact us.



—



Sehr verehrte Kunden,



nach der Core Router Maintenance von heute Morgen haben wir vereinzelt Meldungen ??ber nicht erreichbare IP Netze. Dies wird zur Zeit von uns gepr??ft.



Sollten sie Probleme bei der Erreichbarkeit feststellen m??chten wir sie bitten uns sog. Traceroutes von/zu ihrem Server bei uns zu senden. Basierend auf diesen Daten kann eine Analyse erfolgen.



Mit freundlichen Gr??ssen



LeaseWeb