Category Archives: Leaseweb

Leaseweb storing? Actuele Leaseweb storingen en onderhoud

15 jul

emergency router software upgrade

Startdate: July 16th 2009 00:01:00(CEST)
Enddate: July 16th 2009 01:00:00(CEST)

Dear Customer,

We have scheduled an emergency software upgrade of one router.
The software upgrade is needed to resolve some possible bugs with network latency.

The reload of the router should take 5 to 10 minutes.

Affected IP ranges:
62.212.84.0/24
94.75.216.0/23
94.75.220.0/22
94.75.229.0/24
94.75.240.0/24
94.75.246.0/24
94.75.252.0/23
95.211.92.0/23
94.75.234.0/24

Kind regards,

LeaseWeb NOC

15 jul

EvoSwitch issues [Update]

Time: July 15th 2009 12:30:00(CEST)
Location: EvoSwitch
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network located at EvoSwitch.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 12:40] The problem has been solved, we are still investigating the cause of this issue.

[UPDATE 12:43] There are still small parts of our network offline, we are doing our utmost to bring everything online as soon as possible.

[UPDATE 12:50] All connectivity has been restored, if you still experience any issues please contact us by phone at +31203162880. We are still investigating the cause of this issue.

15 jul

EvoSwitch issues [Update]

Time: July 15th 2009 12:30:00(CEST)
Location: EvoSwitch
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network located at EvoSwitch.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 12:40] The problem has been solved, we are still investigating the cause of this issue.

[UPDATE 12:43] There are still small parts of our network offline, we are doing our utmost to bring everything online as soon as possible.

[UPDATE 12:50] All connectivity has been restored, if you still experience any issues please contact us by phone at +31203162880. We are still investigating the cause this issue.

13 jul

Telia maintenance [Update]

Startdate: July 14th 2009 06:00:00(CEST)
Enddate: July 14th 2009 07:30:00(CEST)

Dear customer,

Our carrier Telia announced a maintenance on most of our uplinks with them.

During this maintenance customers may experience temporarily higher latency on some paths.

We don’t expect any downtime as our core routers will reroute traffic towards our other carriers.

Best regards,

LeaseWeb – NOC

Timezone is CEST


11 jul

AMS-IX Telecity ports down [Update]

Time: July 8th 2009 16:23:00(CEST)
Expected Downtime: Unknown
Reason: Under investigation / Unknown
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 16:35]
The ports to the AMS-IX switches came up after 7 minutes downtime.
We are investigating the cause.
The AMS-IX NOC has also been contacted, they are investigating their side.

When we receive an update we will update this article.

[UPDATE]

We have received the following from the AMS-IX NOC.

—–
Ticket: 24514
Summary: Reboot of pxc-tel-115
Status: OPEN
Opened: 2009-07-08 16:48 +0200
Type: unscheduled
Scope: AMS-IX
Start: 2009-07-08 16:17 +0200
End: 2009-07-08 16:25 +0200

DESCRIPTION:

The photonic switch pxc-tel-115 rebooted, thereby dropping all its connections.

DETAILS:

During this service interruption there was maintenance work going on in the suite where this PXC is located. Work on the power feed for other devices may have triggered this outage.

ACTIONS:

Once the PXC was active again, the AMS-IX NOC restored connectivity for the affected members.

We are investigating the cause of the reboot.

AFFECTED:

The following members saw an outage of 8 minutes on their 10GE connection:

ZeelandNet BV 15542 195.69.145.43
NPO 25182 195.69.145.86 2001:7f8:1::a502:5182:1
UPC Broadband Operation 6830 195.69.144.174
Facebook, Inc 32934 195.69.145.115
Versatel AG 8881 195.69.145.109 2001:7f8:1::a500:8881:1
KPN Telecom B.V. (ISP I 8737 195.69.144.84 2001:7f8:1::a500:8737:2
Leaseweb 16265 195.69.144.215 2001:7f8:1::a501:6265:2
—–

11 jul

SSC not accessible [Update]

Time: July 3rd 2009 16:30:00(CEST)
*******************************

Dear Customer,

At this moment the SSC (secure.leaseweb.com) is down for maintenance

We will update this post when we have an ETR.

Regards,
LeaseWeb BV


[UPDATE]

Dear customer,

Due to a vulnerability attack and as precautionary followup measure, we will not power up the Self Service Center (SSC) for the time being. We kindly ask you to contact our support at [email protected] if you need to manage your services during that time. We will keep you posted on this site.

[UPDATE Sun Jul 5th 21:20]

On Friday we sent you a security notice on a breach of our systems. We are happy to inform you that we could contain the situation and that everything is under full control at Leaseweb.

Our precautionary measures have been lifted, so you can use the Self Service Center (SSC) again as usual. To be safe, we still advise you to change the passwords to your services as soon as possible. Events have shown that no one is safe from hacker.s attacks!

Assisted by the renowned IT Security firm FOX-IT, we have been implementing further security measures and are now investigating the complete backgrounds of the attack. Minor Leaseweb systems or parts thereof are still offline to this end but very shortly all systems will be completely back to normal at Leaseweb.

We apologize again for the inconvenience our stringent precautions may have been causing.

Best regards,
Your Leaseweb Team

11 jul

Power issue Evoswitch hall3 [Update]

Time: July 2nd 2009 21:40:00(CEST)
Expected Downtime: Unknown
Location: EvoSwitch Hall3
Reason: Power failure / Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 21:55]
The power issue has (partially) affected three rows of racks.
Datacenter engineers are currently troubleshooting the problem.

[UPDATE 22:25]
The datacenter engineers reported they expect to restore power within 10 minutes.

[UPDATE 22:41]
The datacenter engineers reported a defect in a power circuit.

[UPDATE 22:46]
New ETR is 10 minutes

[UPDATE 22:58]
There are some difficulties repairing the issue
New ETR will be posted as soon as we get confirmation.

[UPDATE 23:15]
Power has been restored.
Engineers are checking all affected racks for servers that do not come back online.

[UPDATE]
Some previous incorrect posts have been editted.

11 jul

link down [Update]

Time: June 27th 2009 11:45:00(CEST)
Expected Downtime: Unknown
Reason: Hardware failure / Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.
A link has failed between one of our routers and an MPLS device.
Because of this most of our DSL customers are no longer able to reach the internet.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV


[UPDATE 12:30]
The engineer is on his way to the datacenter with all spare parts he may need.

[UPDATE 13:15]
Our engineer replaced broken optic module at our MPLS device and the failed connection went up. We believe the issue is solved, however we are doing some tests to fully confirm it. If we find anything wrong we will keep you informed.

09 jul

AMS-IX Telecity ports down [Update]

Time: July 8th 2009 16:23:00(CEST)
Expected Downtime: Unknown
Reason: Under investigation / Unknown
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 16:35]
The ports to the AMS-IX switches came up after 7 minutes downtime.
We are investigating the cause.
The AMS-IX NOC has also been contacted, they are investigating their side.

When we receive an update we will update this article.

[UPDATE]

We have received the following from the AMS-IX NOC.

—–
Ticket: 24514
Summary: Reboot of pxc-tel-115
Status: OPEN
Opened: 2009-07-08 16:48 +0200
Type: unscheduled
Scope: AMS-IX
Start: 2009-07-08 16:17 +0200
End: 2009-07-08 16:25 +0200

DESCRIPTION:

The photonic switch pxc-tel-115 rebooted, thereby dropping all its connections.

DETAILS:

During this service interruption there was maintenance work going on in the suite where this PXC is located. Work on the power feed for other devices may have triggered this outage.

ACTIONS:

Once the PXC was active again, the AMS-IX NOC restored connectivity for the affected members.

We are investigating the cause of the reboot.

AFFECTED:

The following members saw an outage of 8 minutes on their 10GE connection:

ZeelandNet BV 15542 195.69.145.43
NPO 25182 195.69.145.86 2001:7f8:1::a502:5182:1
UPC Broadband Operation 6830 195.69.144.174
Facebook, Inc 32934 195.69.145.115
Versatel AG 8881 195.69.145.109 2001:7f8:1::a500:8881:1
KPN Telecom B.V. (ISP I 8737 195.69.144.84 2001:7f8:1::a500:8737:2
Leaseweb 16265 195.69.144.215 2001:7f8:1::a501:6265:2
—–

09 jul

Telia maintenance

Startdate: July 14th 2009 06:30:00(CEST)
Enddate: July 14th 2009 07:30:00(CEST)

Dear customer,

Our carrier Telia announced a maintenance on most of our uplinks with them.

During this maintenance customers may experience temporarily higher latency on some paths.

We don’t expect any downtime as our core routers will reroute traffic towards our other carriers.

Best regards,

LeaseWeb – NOC

Timezone is CEST