Yesterday evening, on Tuesday 28th February, Amazon’s web-based S3 storage services experienced issues worldwide. These issues included it being completely or partially broken for both apps and devices utilising it. Images on websites as well as whole websites and the backend of apps such as Nest are hosted by Amazon SWS S3, so much of these were affected with the problems yesterday.
About AWS S3
Amazon’s S3 web-based storage service, AWS, plays a role in hosting over 148,000 websites, including nearly 122,000 unique domains. It is particularly widely used within the USA as a content hosting platform.
What caused the outage?
The outage was caused by “high error rates within S3 in US-EAST-1” as per the company’s official update dashboard, which promised that resolutions were being worked on from the moment that the issues occurred around 7:17pm.
Well-known websites and services that were affected
Some of the well-known websites and services that have been affected by the outage include Giphy, Business Insider, Sailthru and Quora, just to name a few. Image hosting was also affected for across a wide range of websites, including a range of publishing websites, as well as file sharing sites like Slack. One of the common issues was the dashboard not changing colour, which was directly related to the S3 issue.
Updates about the issues with Amazon AWS S3
Throughout the downtime, Amazon regularly updated their help forum to keep customers affected in the loop. Around 11:40pm, Amazon updated their dashboard to say that dashboard related issues had been fixed, and updates will now show accurately the status of all services. By 11:57 they updated this dashboard to give the further information that they had an understanding of the “root cause” of the S3 issues and work working to get these resolved, although no further details were given.
Improvements throughout the night
By 12:15pm, ThousandEyes, the network intelligence software provider, had noted that the Ashburn, VA area was primarily where the ongoing packet loss issue was occurring. There had been a fire there last year during the construction of a new Amazon AWS centre there, which had given away this location. At 12:54pm, some avatars, objects and visual aspects were beginning to be restrored as AWS saw “recovery for S3 object retrievals, listings and deletions”, and further fixes were expected to happen within the following hour.
Full recovery of Amazon’s AWS S3 services
From 1:20pm, full recovery for S3 was reported in terms of objects being retrieved, listed and deleted. The status page reported that normal operation would soon be fully restored for the S3-based storage use. At 2:10pm, all errors were reported to be fully resolved, and S3 services were back to “operating normally”.