FASTLY -- Global CDN Disruption

Incident Report for Scaleflex Visual Experience Platform

Postmortem

FASTLY - Global CDN Disruption (Jun-8, 2021)

‌

FASTLY experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95% of our network was operating as normal.

This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them.

‌

You can find a full summary on the FASTLY Blog

Posted Jun 10, 2021 - 11:34 CEST

Resolved

Dear Customers,

Please find hereunder the latest feedback from FASTLY

Jun 8, 12:41 UTC | Resolved - Fastly has observed recovery of all services and has resolved this incident. Customers could continue to experience a period of increased origin load and lower Cache Hit Ratio (CHR).

This incident is now closed.

Regards,

Scaleflex Support team

Posted Jun 08, 2021 - 16:10 CEST

Update

We are continuing to monitor for any further issues.

Posted Jun 08, 2021 - 16:09 CEST

Monitoring

Dear Customers (FASTLY users only),

Please be aware that FASTLY status page is back online [https://status.fastly.com].

-----------------------------------
FASTLY - Global CDN Disruption
-----------------------------------

Jun 8, 10:57 UTC | Monitoring - The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.
Jun 8, 10:57 UTC | Monitoring - The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.
Jun 8, 10:44 UTC | Identified - The issue has been identified and a fix is being implemented.
Jun 8, 10:26 UTC | Update - We are continuing to investigate this issue.
Jun 8, 10:23 UTC | Update - We are continuing to investigate this issue.
Jun 8, 10:21 UTC | Update - We are continuing to investigate this issue.
Jun 8, 10:11 UTC | Update - We are continuing to investigate this issue.
Jun 8, 10:07 UTC | Update - We are continuing to investigate this issue.
Jun 8, 09:58 UTC | Investigating - We're currently investigating potential impact to performance with our CDN services.

We will monitor the service on our side.

Regards,

Scaleflex Support team

Posted Jun 08, 2021 - 13:04 CEST

Identified

Dear Customers,

If you are using the FASTLY network, you may encounter a strong degradation of services.
Unfortunately FASTLY has not communicated yet on the outage and we have no visibility on when the FASTLY service will be recovered.

Scaleflex Support team is working actively to mitigate the situation.

We will update you as soon as we have more visibility.

Regards,

Scaleflex Support team

Posted Jun 08, 2021 - 12:44 CEST

This incident affected: Dynamic Media Optimization (Media Delivery CDN).