16 dec

Performance issues: [RESOLVED] Power issues in a single availability zone

Dec 9 1:08 AM PST We are investigating connectivity issues for instances in the US-EAST-1 region.
1:26 AM PST We are experiencing power issues for a subset of instances in a single availability zone in the US-EAST-1 region.
1:51 AM PST The underlying power issue has been addressed. Instances have begun to recover.
2:11 AM PST Most affected instances have regained power and are operating normally. We are working to recover a small number of instances that have not yet recovered.
3:33 AM PST We are continuing to work on recovering the remaining affected instances.
5:31 AM PST We are continuing to work on recovering the small number of remaining affected instances.
6:41 AM PST We have completed the recovery of most instances affected by this event. A very small number of instances are still not responding. We will continue to attempt to restore all instances or notify users directly if the instances remain degraded.

Dec 15 12:08 PM PST (revision of Dec 10 post to clarify timing) We would like to provide further information on the issue experienced on December 9 in one of our east coast availability zones (this event affected a minority of instances in one of our four Availability Zones in the US-EAST-1 Region). A single component of the redundant power distribution system failed in this zone. Prior to completing the repair of this unit, a second component, used to assure redundant power paths, failed as well, resulting in a portion of the servers in that availability zone losing power. Impacted customers experienced a loss of connectivity to their instances. As soon as the defective power distribution units were bypassed, servers restarted and instances began to come online shortly thereafter. Over 25% of these instances recovered within 30 minutes, and over 90% recovered within an hour. A small number of instances took up to a few hours to recover, and we worked with those customers during the morning.